{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PPO, Actor-Critic Style\n",
    "_______________________\n",
    "&nbsp;&nbsp;**for** iteration=1,2,... do<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**for** actor=1,2,...,N do<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Run policy $\\pi_{\\theta_{old}}$ in environment for T timesteps<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute advantage estimates $\\hat{A}_1,\\dots,\\hat{A}_T$<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**end for**<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Optimize surrogate(代理人) L wrt $\\theta$,with $K$ epochs and minibatch size $M \\leq NT$<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\\theta_{old} \\leftarrow \\theta$<br>\n",
    "&nbsp;&nbsp;**end for**\n",
    "_______________________"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss Function L的数学公式为:\n",
    "$$\n",
    "L_t^{CLIP+VF+S}(\\theta)=\\hat{\\mathbb{E}_t}[L_t^{CLIP}(\\theta)-c_1L^{VF}_t(\\theta)+c_2S[\\pi_\\theta](s_t)]\n",
    "$$\n",
    "其中，$L^{CLIP}(\\theta)=\\hat{\\mathbb{E}_t}\\big[min(r_t(\\theta)\\hat{A}_t,clip(r_t(\\theta), 1-\\epsilon,1+\\epsilon)\\hat{A}_t)\\big]$, $r_t(\\theta)=\\frac{\\pi_\\theta(a_t|s_t)}{\\pi_{\\theta_{old}}(a_t|s_t)}$<br>\n",
    "$L^{VF}_t=(V_\\theta(s_t)-V_t^{targ})^2$ **critic loss**<br>\n",
    "S 为奖励熵，保证足够多的探索(写A2C的时候已经OK)<br>\n",
    "$c_1, c_2$为参数\n",
    "#### $L^{CLIP}和r的关系如下(为了保证\\pi_\\theta和\\pi_{\\theta_{old}}的差值不会很大,满足TRPO中两者方差变化不大的要求)$：\n",
    "<img src=\"../assets/PPO_CLIP.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GAE(high-dimensional continuous control using Generalized Advantage Estimation)\n",
    "We address the first challenge by using value functions to substantially reduce the variance of policy gradient estimates at the cost of some bias, with an exponentially-weighted estimator of the advantage function that is analogous to TD(λ). <br>\n",
    "\n",
    "改进了advantage function的计算方式。将advantage function进行类似于TD(λ)的处理<br>\n",
    "\n",
    "#### 推导过程\n",
    "1. 原始的advantage function : $\\delta^V_t=r_t+\\gamma V(s_{t+1})−V(s_t)$\n",
    "2. $在位置t时,其后k个 \\delta 折扣相加$ : \n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\hat{A}^{(1)}_t&:=\\delta^V_t&&=-V(s_t)+r_t+\\gamma V(s_{t+1}) \\\\\n",
    "\\hat{A}^{(2)}_t&:=\\delta^V_t+\\gamma \\delta^V_{t+1}&&=-V(s_t)+r_t+\\gamma r_{t+1}+\\gamma ^2 V(s_{t+2}) \\\\\n",
    "\\hat{A}^{(3)}_t&:=\\delta^V_t+\\gamma \\delta^V_{t+1}+\\gamma^2 \\delta^V_{t+2}&&=-V(s_t)+r_t+\\gamma r_{t+1}+\\gamma^2 r_{t+2}+\\gamma ^3 V(s_{t+3}) \\\\\n",
    "\\hat{A}_t^{(k)}&:=\\sum_{l=0}^{k=1}\\gamma^l\\delta_{t+l}^V&&=-V(s_t)+r_t+\\gamma r_{t+1}+\\dots+\\gamma^{k-1}r_{t+k-1}+\\gamma^kV(s_{t+k})\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "\n",
    "3. $k \\to \\infty, \\gamma^kV(s_{t+k})$会变得非常非常非常小，So :\n",
    "$$\n",
    "\\hat{A}_t^{(\\infty)}=\\sum^\\infty_{l=0}\\gamma^l\\delta_{t+l}^V=-V(s_t)+\\sum^\\infty_{l=0}\\gamma^lr_{t+l}\n",
    "$$\n",
    "4. 所以，$t$ 时刻的GAE可推导为 :\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\hat{A}_t^{GAE(\\gamma, \\lambda)}&:=(1-\\lambda)\\big(\\hat{A}_t^{(1)}+\\lambda\\hat{A}_t^{(2)}+\\lambda^2\\hat{A}_t^{(3)}+\\dots\\big)\\\\\n",
    "&=(1-\\lambda)\\big(\\delta_t^V+\\lambda(\\delta_t^V+\\gamma\\delta_{t+1}^V)+\\lambda^2(\\delta_t^V+\\gamma\\delta_{t+1}^V+\\gamma^2\\delta_{t+2}^V)+\\dots\\big)\\\\\n",
    "&=(1-\\lambda)\\big(\\delta^V_t(1+\\lambda+\\lambda^2+\\dots)+\\gamma\\delta^V_{t+1}(\\lambda+\\lambda^2+\\lambda^3+\\dots)+\\gamma^2\\delta^V_{t+2}(\\lambda^2+\\lambda^3+\\lambda^4+\\dots)+\\dots\\big)\\\\\n",
    "&=(1-\\lambda)\\big(\\delta^V_t\\big(\\frac{1}{1-\\lambda}\\big)+\\gamma\\delta^V_{t+1}\\big(\\frac{\\lambda}{1-\\lambda}\\big)+\\gamma^2\\delta^V_{t+2}\\big(\\frac{\\lambda^2}{1-\\lambda}\\big)+\\dots\\big)\\\\\n",
    "&=\\underbrace{\\delta^V_t+\\gamma\\lambda\\delta^V_{t+1}+(\\gamma\\lambda)^2\\delta^V_{t+2}+\\dots}_{此处计算时使用这个公式(迭代计算)}\\\\\n",
    "&=\\sum_{l=0}^\\infty(\\gamma\\lambda)^l\\delta^V_{t+l}\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用高斯分布(正态分布)来实现随机性策略控制连续动作空间\n",
    "1. 高斯分布有两个重要的变量一个是均值 $\\mu$ ,另一个是方差 $\\sigma$ 。$\\mu$ 为高斯函数的对称轴，$\\frac{1}{\\sqrt{2\\pi}\\sigma}$ 为高斯函数的最高点。高斯函数的积分为1。所以我们可以使用它来进行连续动作的sample。方差 $\\sigma$ 越大，分布越分散，方差 $\\sigma$ 越小，分布越集中。\n",
    "2. $\\mu$ 的选择很好把控，经过tanh处理之后+简单的数学变换，使nn输出的 $\\mu$ 在env规定的动作空间内就可以\n",
    "3. $\\sigma$ 的选择,使用softplus函数对sigma进行处理。softplus 公式为$f(x)=\\frac{1}{\\beta}log(1+exp(\\beta x))$, softplus 是 ReLU 的平滑近似值版本\n",
    "4. 高斯分布公式:\n",
    "$$\n",
    "f(x)=\\frac{1}{\\sqrt{2\\pi}\\sigma}exp\\bigg(-\\frac{(x-\\mu)^2}{2\\sigma^2}\\bigg)\n",
    "$$\n",
    "5. 和确定性策略相比，需要考虑每个state采取每个动作的概率，计算量确实比较大。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TRPO\n",
    "简单理解为一次on-policy到off-policy的转换<br>\n",
    "但是为了保证old_policy和new_policy之间方差相差不会太大<br>\n",
    "$$\n",
    "\\begin{aligned}\n",
    "E_{X \\sim p}[f(x)] & \\approx \\frac{1}{N}\\sum^N_{i=1}f(x^i)\\\\\n",
    "&= \\int f(x)p(x)dx=\\int f(x)\\frac{p(x)}{q(x)}q(x)dx=E_{x \\sim q}[f(x)\\frac{p(x)}{q(x)}]\n",
    "\\end{aligned}\n",
    "$$\n",
    "由此，在两者期望相同的情况下，论证方差是否相同\n",
    "$$\n",
    "\\begin{aligned}\n",
    "两者期望:\\quad&\\because E_{X \\sim p}[f(x)]=E_{x \\sim q}[f(x)\\frac{p(x)}{q(x)}]\\\\\n",
    "方差公式:\\quad&\\because VAR[X]=E[X^2]-(E[X])^2\\\\\n",
    "x \\sim p 方差:\\quad&\\therefore Var_{x \\sim p}[f(x)]=\\color{red}{E_{x\\sim p}[f(x)^2]}-(E_{x\\sim p}[f(x)])^2\\\\\n",
    "x \\sim q 方差:\\quad&\\therefore Var_{x \\sim q}[f(x)\\frac{p(x)}{q(x)}]=E_{x \\sim q}\\big[\\big([f(x)\\frac{p(x)}{q(x)}\\big)^2\\big]-\\big(E_{x\\sim q}\\big[f(x)\\frac{p(x)}{q(x)}\\big]\\big)^2\\\\\n",
    "&=\\color{red}{E_{x \\sim q}\\big[f(x)^2\\frac{p(x)}{q(x)}\\big]}-(E_{x \\sim p}[f(x)])^2\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "两者方差公式的差别在标红的位置，也就是说我们如果使两者$E_{x\\sim p}[f(x)^2]$和$E_{x \\sim q}\\big[f(x)^2\\frac{p(x)}{q(x)}\\big]$的差值较小，那么我们所做的off-policy就是可行的<br>\n",
    "由此，可直观的看出，我们要使p(x)和q(x)的相差较小。因此就有了PPO1中的所使用的$\\beta KL(\\theta,\\theta')$和PPO2中的clip这些都是为了限制两者的范围在一个可接受的合适空间"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torch.distributions import Normal\n",
    "from torch.distributions import Categorical\n",
    "import torch.multiprocessing as mp\n",
    "# from torch.utils.tensorboard import SummaryWriter\n",
    "import numpy as np\n",
    "from IPython.display import clear_output\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import math\n",
    "import random\n",
    "from statistics import mean\n",
    "import pdb\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAADzCAYAAACsa+1AAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdeXxU5dXA8d/JDoEESMKWBAKEfZVdxH2DgqACKi7F1ta2LvWtvlbtomCtrX2t1lq17gsuCLiBYhGl4oIiYSdshrAkhEAIZCN7ct4/5mLTGJIBktyZyfl+Pvkwc+9z7z03ZGbOPPe5zxFVxRhjjDHGlwS5HYAxxhhjTG2WoBhjjDHG51iCYowxxhifYwmKMcYYY3yOJSjGGGOM8TmWoBhjjDHG51iCYr4jIh+KyKxmOE6SiKiIhDT2cUXkTBHZXuP5bhG5oDH27ewvVUTOaaz9GWO+T0QuE5EMESkSkdNOYvvZIvJqU8Rmmo8lKC4SkatEZJWIHBWRg87jm0RE3IhHVSeq6su+elwnqUluYF+fq2rfxohLRF4SkQdq7X+gqn7aGPs3JtCJyHgRWSki+SJyWES+FJFRXmz6MHCLqrZR1XWN/UXD+AdLUFwiIncAjwH/B3QGOgE/B84AwlwMzW8d65ExxrhPRKKA94HHgQ5APDAHKPNi8+5AatNFZ/yBJSguEJFo4H7gJlVdqKqF6rFOVa9R1TKn3SQRWSciBU535+wa+zhHRDJr7fe7bxkiMlpEUpxtD4jII87yCBF5VURyRSRPRFaLSCdn3aci8hPncS8RWe60OyQir4lIu1rH+l8R2eh8O3pTRCKOc77BIvKws590YFKt9TWPmywiK5x9HhKRN53lnznNNzjdvlce+x2IyF0ikg28WNfvBRglIltE5IiIvHgsThG5XkS+qBWLOjHcCFwD/No53uI6fsfhIvI3Eclyfv4mIuE1/39E5A6nd2y/iPzoOH8SxgSiPgCq+oaqVqlqiap+pKobRSRIRH4nInuc18crIhLtvKaKgGA8r/WdIjIX6AYsdl6Lv5b/XCa+0Xnt7Xe+9H3Pyb5XGvdZguKO04Fw4L0G2h0Ffgi0w/Oh/gsRudTLYzwGPKaqUUAvYL6zfBYQDSQCMXh6bUrq2F6APwFdgf5O+9m12lwBTAB6AEOA648Ty0+BycBpwEhgej1x/wH4CGgPJOD59oWqnuWsH+p0+77pPO+M59tZd+DG4+zzGuBiPL+HPsDv6jk+zvGeAV4D/uIc75I6mv0WGAsMA4YCo2vtuzOe33U8cAPwhIi0b+jYxgSIHUCViLwsIhNr/e1f7/ycC/QE2gD/UNUyVW3jtBmqqr1U9TpgL3CJ81r8S439nAv0Bi4C7j7Jy0DHe680LrMExR2xwCFVrTy2wLlOmyciJSJyFoCqfqqqm1S1WlU3Am8AZ3t5jAogWURiVbVIVb+usTwGSHa+1axR1YLaG6tqmqouc94wcoBH6jj231U1S1UPA4vxfFDX5Qrgb6qa4bT9UwNxdwe6qmqpqn5RT1uAauA+J866Ei3wvPEdO/YfgZkN7NNb1wD3q+pB53c0B7iuxvoKZ32Fqi4BioBGGR9jjK9z3lfGAwo8C+SIyCKnx/Ya4BFVTVfVIuAe4KqTuEw7R1WPquom4EVO7rV9vPdK4zJLUNyRC8TWfDGq6jhVbeesCwIQkTEi8m8RyRGRfDy9HbFeHuMGPL0F25zLOJOd5XOBpcA8p2v0LyISWntjEekoIvNEZJ+IFACv1nHs7BqPi/F8C6pLVyCjxvM99cT9azy9N9+I546ZH9fTFiBHVUsbaFP72F0baO+trvz3udTed27NJJT6f0fGBBxV3aqq16tqAjAIz+vjb9T92gnBMxbvRDTGa/t475XGZZaguOMrPAPFpjbQ7nVgEZCoqtHAP/F8eIPn8k/rYw1FJBiIO/ZcVb9V1ZlAR+AhYKGIRDrf5ueo6gBgHJ5LLz+s49h/wvPNZ4jT9XltjWOfqP14LhEd0+14DVU1W1V/qqpdgZ8BT0r9d+54U4679rGznMe1f4edT3DfWXh6e+ratzGmBlXdBryEJ1Gp67VTCRw43ubHWX6813ZNJ/VeWd+5mOZhCYoLVDUPz+WAJ0Vkuoi0cQaNDQNqvjDaAodVtVRERgNX11i3A4gQz0DaUDxjH8KPrRSRa0UkTlWrgTxncZWInCsig50XaQGe7s2qOsJsi+eSRJ6IxAN3nsIpzwd+KSIJznXou4/XUERmiEiC8/QInjemY/EdwHO9+kTd7By7A/Ab4Nj4lQ3AQBEZ5gycnV1ru4aO9wbwOxGJE5FY4F48PU3GtHgi0s8ZJJ7gPE/EcwnmazyvnV+JSA8RaQM8CLxZq8expuO9Fn8vIq1FZCDwI/7z2q7ppN4rT/iETaOzBMUlzkCv2/Fc0jiI5wX4NHAXsNJpdhNwv4gU4vnwm19j+3xn/XPAPjzfEmqOVJ8ApDoj4h8DrnIuhXQGFuJJTrYCK6j7Q3UOMBzIBz4A3j6F030Wz2WlDcDaBvY1CljlxL0IuE1VdznrZgMvO2N1rjiB47+OZ+BtuvPzAICq7sBzN9XHwLdA7fEuzwMDnOO9W8d+HwBSgI3AJufcHqijnTEtUSEwBs/r+SiexGQzcAfwAp7LzZ8Bu4BS4NZ69vUnPF8G8kTkf2ssXwGkAZ8AD6vqR7U3PIX3SuMyUfWmh9wYY4zxDSKShCexCa2n18X4OetBMcYYY4zPsQTFGGOMMT7HLvEYY4wxxudYD4oxxhhjfI4lKMYYY4zxOX5V/TU2NlaTkpLcDsMYA6xZs+aQqsY13NK32PuIMb6jvvcRv0pQkpKSSElJcTsMYwwgIvWVLPBZ9j5ijO+o733ELvEYY4wxxudYgmKMMcYYn+NVgiIiE0Rku4ikicj36qiISLiIvOmsX+XM8ldzfTcRKao5RXFD+zTGmGNE5AUROSgim2ss6yAiy0TkW+ff9m7GaIxpXA0mKE5RuSeAicAAYKaIDKjV7AbgiKomA4/iqQhZ06PAhye4T2OMOeYlPDVTarob+ERVe+OpxWJfdIwJIN70oIwG0lQ1XVXLgXnA1FptpgIvO48XAueLiACIyKV4CrSlnuA+jTEGAFX9DDhca3HN952XgUubNShjTJPyJkGJBzJqPM90ltXZxinclA/EiEgknuq8c05in8YYU59OqrofwPm3o8vxGGOAxpqh3pvbjKWu43vZZg7wqKoWOR0qJ7JPT0ORG4EbAbp169ZgsMZ35BSWsWlfHluyCsjKL+VgQRlllVVUVSutw0KIbhVKQvtW9IyLZFhiO7p1aE2tvxNjGoW9jxjTPLLzS/nRS6v542WDGN7t1IaFeZOgZAKJNZ4nAFnHaZMpIiFANJ7u2DHAdBH5C9AOqBaRUmCNF/sEQFWfAZ4BGDlypBUO8mHV1cqavUdYujmbZVsPsCe3+Lt1HSLD6Ng2nFZhwQSLcKS4hNSsfLILSjmWbHeJjuD0XjFMHtKFM3vHERpsN5mZeh0QkS6qul9EugAHj9fQ3keMaR5//nArO3OKiI0MP+V9eZOgrAZ6i0gPYB9wFXB1rTaLgFnAV8B0YLl6+njOPNZARGYDRar6DyeJaWifxk/kFZezICWTuV/vYe/hYsKCgxiXHMN1Y7szJKEdA7tGERle959aaUUVuw4dZc2eI3ydnsvybQd5e+0+OkSGMW14PD8e34Mu0a2a+YyMnzj2vvNn59/33A3HmJZtzZ7DvLs+i5vP7UW3mNanvL8GExRVrRSRW4ClQDDwgqqmisj9QIqqLgKeB+aKSBqenpOrTmafp3guppkVlFbwzIp0nv9iFyUVVYxKas/tF/bh/P4daRsR6tU+IkKD6d8liv5dorh2bHfKK6tZsSOHd9Zl8sKXu3nxy91MHRbP/1zQm8QOp/4Hb/yTiLwBnAPEikgmcB+exGS+iNwA7AVmuBehMS1bdbUyZ/EWOkWFc9M5yY2yT2mswSzNYeTIkWpTVLuvqlp5eeVu/r78W/KKK5g8pAu/OKcXA7tGN+pxMg4X8/wXu5i3ei/VCj8Z34Obzk2mzXF6Y0zzEpE1qjrS7ThOlL2PGNP45q/O4NdvbeRvVw7j0tO8v+elvvcRe6c3J2TzvnzueXsTm/blc2bvWO6a0I9B8Y2bmByT2KE1s6cM5Odn9+IvS7fx5Kc7eWttJn++fAjn9rMbNowxxhcUlFbwl6XbGNG9PVOHdW20/dooROOVqmrl8U++ZeoTX7I/v5R/XH0ar/x4dJMlJzV1jo7gkSuG8e7NZ9CuVRg/emk1dy3cSGFpRZMf2xhjTP0e/+Rbco+WM/uSgY16J6b1oJgGHSwo5X/eXM/KnblMGdqVP0wdRHRr78aYNKZhie1YdOsZ/O3jb3l6xU6+2X2Yf147gr6d2zZ7LMYYY2BnThEvfrmbK0YkMjihcb+wWg+Kqdf6jDwmPf4Fa/ce4S/ThvDYVcNcSU6OCQ8J5q4J/Xjjp2MpKqvk0ie+5L31+1yLxxhjWrI/vL+FVqHB3Dmhb6Pv2xIUc1yLN2Rx5dNfEREaxHs3j+eKUYk+M5HamJ4xfHDreAbFR3HbvPU8umxHo81eaIwxpmHLtx3g0+053HZBb2LbnPq8J7VZgmLq9M8VO7n1jXUMjo/m3ZvO8MnLKB2jInj9p2OZPiKBxz75ll8v3EhFVbXbYRljTMArr6zmD+9vpWdcJD88PalJjmFjUMx/UVUeWbaDx5enccnQrjw8YwjhIcFuh3VcocFB/N/0IXRt14q/OwO1nrxmOBGhvhuzMcb4uxe/3MWuQ0d56UejCAtpmr4O60Ex31FVHvhgK48vT+PKkYn87cphPp2cHCMi3H5hHx64dBDLtx3kxrlrKK2ocjssY4wJSAcLS3l8eRrn9+vIOX2bbsoHS1AM4ElO7luUyvNf7OL6cUn86fLBBAf5xngTb107tjt/mTaEz7/N4Scvp1BSbkmKMcY0tr/8aztllVX8bvKAJj2OJSgGgL9+tINXvtrDjWf15L5LBhDkZ8nJMVeMSuTh6UP5cuchbn59rY1JMcaYRrQ+I4+FazL58fge9IiNbNJjWYJiePazdP7x7zRmjk7knon9fOZOnZM1bUTCd5d77lq4kepqu7vHGGNOVXW1MntRKnFtw7n1vN5NfjwbJNvCzU/J4I9LtjJpcBceuHSw3ycnx1wzpjuHi8r567IddIgM47eT+gfMuRljjBveWbeP9Rl5PDxjaLPURLMEpQVbmXaI37y9iTN7x/LolcP8bsxJQ245L5nco+U898UuOkdH8JMze7odkjHG+KWiskr+/K9tDE1sx+UnUAzwVFiC0kKl5xTx81fX0DMukieuGd5kt4m5SUS4d/IADhSU8uCSrfSKa2NFBo0x5iT8Y3kaOYVlPHPdiGYbo+jVp5KITBCR7SKSJiJ317E+XETedNavEpEkZ/loEVnv/GwQkctqbPMrEUkVkc0i8oaIRDTWSZn65RWXc8PLKYQEB/H8rFFERbg3dX1TCwoS/nrFUPp1juKXb6wj7WCh2yEZY4xf2X3oKC98sYtpwxM4rVv7ZjtugwmKiAQDTwATgQHATBGpfW/RDcARVU0GHgUecpZvBkaq6jBgAvC0iISISDzwS2fdICAYuKoxTsjUr7KqmpteW8u+IyU8c90IEju0djukJtc6LIRnZ40kPDSYG15O4cjRcrdDMsYYv/HAB1sIDRbuaoJ6O/XxpgdlNJCmqumqWg7MA6bWajMVeNl5vBA4X0REVYtVtdJZHgHUvJ0iBGglIiFAayDrZE/CeO+RZTtYuTOXP142iJFJHdwOp9nEt2vF09eNYH9eKbe9ud7u7DHGGC+s2JHDx1sPcuv5vekY1bwXOrxJUOKBjBrPM51ldbZxEpJ8IAZARMaISCqwCfi5qlaq6j7gYWAvsB/IV9WP6jq4iNwoIikikpKTk+P9mZnv+XjLAZ78dCczRycyY2Si2+E0uxHd23PflAF8tiOHJz9NczscY4zxaRVV1dy/OJWkmNb86IykZj++NwlKXaNhan/9PG4bVV2lqgOBUcA9IhIhIu3x9Lr0ALoCkSJybV0HV9VnVHWkqo6Mi4vzIlxTl4zDxdw+fz0Du0Zx3yUD3Q7HNVeP7sbUYV2dnqRDbodjjDE+6+WVu9mZc5TfTx7gStkTbxKUTKDm1+0Evn855rs2ziWbaOBwzQaquhU4CgwCLgB2qWqOqlYAbwPjTuYETMPKKz3jThR46poRLbqQnojw4GWD6REbyS/fWM/BwlK3QzLGGJ9zqKiMxz7+lrP7xHGeS3c/epOgrAZ6i0gPEQnDM5h1Ua02i4BZzuPpwHJVVWebEAAR6Q70BXbjubQzVkRai2f2rPOBrad8NqZOjyzbwaZ9+fzf9KF0iwn8QbENiQwP4clrRlBUVsGvbDyKMcZ8z8NLt1NSUcXvJw9wbZLLBhMUZ0zJLcBSPEnEfFVNFZH7RWSK0+x5IEZE0oDbgWO3Io8HNojIeuAd4CZVPaSqq/AMpl2LZ2xKEPBMI56XcXydnsvTn3nGnUwY1NntcHxG385tuXfyQL5My+XFlbvdDscYY3zGpsx83kzJ4PpxSSR3bONaHF5N1KaqS4AltZbdW+NxKTCjju3mAnOPs8/7gPtOJFhzYgpKK7hj/ga6d2jN7yY1bdVJfzRzdCKfbD3AQ//axvjkWPp2but2SMYY4ypVZc7iVGIiw/jlBU1fb6c+gTd9qPnOfe+lkl1QyqNXDiOyGeom+BsR4c/ThtA2PIT/eXM9ZZVVbodkjDGuWrQhi5Q9R7jz4r6uT+JpCUqAWrwhi3fW7eOX5/Vu1pn//E1c23AemjaErfsLeGTZDrfDMcYY1xSXV/KnJdsYHB/NjBHuT0VhCUoAyi0q4973NjMssR03n9vL7XB83gUDOnHVqESe/SydDRl5bodjjDGuePLfO8kuKGX2lAHNVm+nPpagBKA5i7dwtKyK/5s+hJBg+y/2xm8m9SeubTh3vbWR8spqt8MxxphmtTe3mGc+T+fSYV0Z0d03Zhm3T68A8/GWAyzakMXN5ybTu5MN+vRWVEQoD1w6mG3ZhTy9Yqfb4ZgTYIVHjTl1f1yyhZAg4e6J/d0O5TuWoASQgtIKfvfuZvp2assvzrFLOyfqwgGdmDykC48vT+PbA1b12B9Y4VFjTt2XaYdYmnqAm89NpnO07+T3lqAEkIc+3MbBwlIemj6EsBD7rz0Zs6cMpHV4MHe9tZEqm8DNX1jhUWNOUmVVNXMWp9KtQ2tuGN/D7XD+i32KBYjVuw/z2qq9/PiMHgxLbOd2OH4rtk04904ewNq9ebz69R63wzENOJHCo8aY73v16z3sOFDEbyf197kyKJagBIDKqmp+/+5m4tu14vaL+rgdjt+77LR4xifH8vBH28kpLHM7HFMPbwuPWlV0Y77v8NFyHlm2g/HJsVw0oJPb4XyPJSgB4JWv9rAtu5DfTx5A6zCbkO1UiQizpwyktKKKh/61ze1wTP28KjxqVdGN+b6/frSdo+VV3HeJe/V26mMJip87WFDKo8t2cHafOC4e6HsZsL9K7tiGn5zZk4VrMknZfbjhDYxbrPCoMSdhS1YBb3yzl+vGdvfZOz4tQfFzf/pwG2WV1cyeMtAnM2B/dut5yXSNjuD376VSWWVzo/giKzxqzIk7Vm8nulUov7rAd4cFWILix1al5/LOun387Oye9IiNdDucgNM6LITfTx7A1v0FNmDWh6nqfaraT1UHqep1qmoDh4ypx5JN2azadZj/vbgv0a3drbdTH68SFBGZICLbRSRNRO6uY324iLzprF8lIknO8tEist752SAil9XYpp2ILBSRbSKyVUROb6yTagkqq6q5b1Eq8e1acdM5yW6HE7AmDOrMmb1j+etHOzhUZJ97xhj/VlJexYNLtjKgSxRXjermdjj1ajBBEZFg4AlgIjAAmCkiA2o1uwE4oqrJwKPAQ87yzXgmUBoGTACeduYqAHgM+Jeq9gOGYteNT8j8lEy2ZRfyu0n9aRXmW7eGBRIR4b5LBlJSUcWjVkzQGOPnnv5sJ/vySpg9ZSDBPlBvpz7e9KCMBtJUNV1Vy4F5eG7rq2kq8LLzeCFwvoiIqharaqWzPAJQABGJAs4CngdQ1XJVtSptXioqq+SRZdsZndSBCYM6ux1OwEvu2IZrx3bnjW/2ssNmmDXG+Kl9eSX8c8VOJg/pwugevlFvpz7eJCjxQEaN55nOsjrbOAlJPhADICJjRCQVzwC2nzvrewI5wIsisk5EnhMRG0Thpac+TeNQUTm/ndTfBsY2k9vO702b8BD++IF19Blj/NODSzzvX7/5ge/U26mPNwlKXZ+AtecAP24bVV2lqgOBUcA9TiGvEGA48JSqngYcBb43tgVsgqXa9uWV8Nznu7h0WFeG2oyxzaZ9ZBi/PL83K3bk8On2g26HY4wxJ+Tr9Fw+2LifX5ydTNd2rdwOxyveJCiZQGKN5wl8v9bFd22cMSbRwH9NHqGqW/EkIoOc9pnOLYLguSw0vK6D2wRL/+3/nInD7pzQz+VIWp7rTu9O95jW/PGDrXbbsTHGb1RVK3MWbyG+XSt+dnZPt8PxmjcJymqgt4j0EJEwPJVCF9VqswiY5TyeDixXVXW2CQEQke5AX2C3qmYDGSLS19nmfGDLKZ5LwFufkce767P4yZk9iPeTDDiQhIcEc8/E/nx7sIg3Vmc0vIExxviAN77Zy9b9BT5Zb6c+DSYozpiRW4CleO60ma+qqSJyv4hMcZo9D8SISBpwO/+5XDMe2CAi64F3gJtU9ZCz7lbgNRHZCAwDHmyskwpEqsofP9hCbJswfmG3Fbvm4oGdGN2jA499/C1Hyyob3sAYY1yUX1zBXz/aztieHZjoZzdVeFW4RVWXAEtqLbu3xuNSYEYd280F5h5nn+uBkScSbEv2ydaDrN59hAcuHUSbcKu34xYR4a4J/Zj21Epe/HIXt5zX2+2QjDHmuB79eAf5JRXcd4n/zTZuM8n6gepq5eGPtpMU05orRyU2vIFpUiO6t+eC/p14ekU6R46Wux2OMcbUaXt2IXO/3sM1Y7rTv0uU2+GcMEtQ/MDijVlsyy7k9ov6Ehps/2W+4M6L+1JUXsk/V+x0OxRjjPkeVeX+91NpEx7C7Rf6br2d+tinnY+rqKrmkWU76N8lismDu7gdjnH07dyWy06L56WVu8nOL3U7HGOM+S9LUw/wZVoud1zUh/aRYW6Hc1IsQfFx81My2JNbzJ0X9yHIx6clbml+dUEfqlV57JNv3Q7FGGO+U1pRxR+XbKFf57ZcPdq36+3UxxIUH1ZaUcXfP/mWEd3bc27fjm6HY2pJ7NCaa8Z0Z35KBrsOHXU7HGOMAeC5z9PJOFzCvZcMIMSPhwX4b+QtwCtf7eZAQRm/vriv342+biluPjeZ8JAg/vrRdrdDMcYY9ueX8MS/d/KDwZ0Z1yvW7XBOiSUoPqqwtIInP93JWX3iGNMzxu1wzHHEtQ3nx2f04P2N+9mebYUEjTHu+tOSbVSr+k29nfpYguKjXvpyN3nFFdx5Ud+GGxtX/eTMHrQJD+HvNhbFGOOib3YdZtGGLH5+di8S2rd2O5xTZgmKDyosreC5L3Zxfr+ODE6Idjsc04B2rcP40RlJfLDJelGMMe6oqlZmL0qla3QEPz+7l9vhNApLUHzQK1/tIb+kgtsusFlK/cUN460XxRjjnnmr97JlfwG/mdSfVmH+U2+nPpag+JijZZU893k65/aNY0hCO7fDMV5q1zqM68clsWSz9aIYY5pXfnEFDy/dzpgeHZgUQPNlWYLiY175ag9Hiiv45fnWe+Jvbhjfg8iwEP6+3HpRjDHN51i9ndlT/K/eTn0sQfEhxeWVPPt5Omf1ieO0bu3dDsecoPaRTi+KjUUxxjQTf6+3Ux9LUHzIq1/v4fDRcm6z3hO/Zb0oxpjmoqrMWezf9Xbq41WCIiITRGS7iKSJyN11rA8XkTed9atEJMlZPlpE1js/G0TkslrbBYvIOhF5vzFOxp+VlFfxzGfpnNk7lhHdrffEX9XsRUk7aL0oxpimszQ1m5U7c/lfP663U58GExQRCQaeACYCA4CZIjKgVrMbgCOqmgw8CjzkLN8MjFTVYcAE4GkRCamx3W3A1lM7hcDw2qo9HCoqt7EnAeDH43sQERLMU5+mux2KMSZAlVZU8Yf3t9Kvc1tm+nG9nfp404MyGkhT1XRVLQfmAVNrtZkKvOw8XgicLyKiqsWqWuksjwD02AYikgBMAp47lRMIBKUVVTz9WTrjesUwKqmD2+GYU9QhMoyrRify3vp9ZB4pdjscY0wAeuazdPbllXDfJQP9ut5Ofbw5q3ggo8bzTGdZnW2chCQfiAEQkTEikgpsAn5eI2H5G/BroPqkow8Qb6/dR05hGbecm+x2KKaR/PTMnojAs59ZL4oxpnHtyyvhyU/TmDS4C6f3CtxSKN4kKHXds6TetlHVVao6EBgF3CMiESIyGTioqmsaPLjIjSKSIiIpOTk5XoTrX6qqlac/28nQhOiA/kNrabq2a8Vlp8Uzb3UGh4rK3A4noIlIOxFZKCLbRGSriJzudkzGNKU/LdmKKtzzg35uh9KkvElQMoHEGs8TgKzjtXHGmEQDh2s2UNWtwFFgEHAGMEVEduO5ZHSeiLxa18FV9RlVHamqI+Pi4rwI1798uHk/e3KL+cU5vQLq/nUDPzu7F+VV1bz45S63Qwl0jwH/UtV+wFBsXJsJYF+n5/L+xv384pzAqLdTH28SlNVAbxHpISJhwFXAolptFgGznMfTgeWqqs42IQAi0h3oC+xW1XtUNUFVk5z9LVfVaxvhfPyKqvLUpzvpGRvJhQM6ux2OaWS94towcVBnXvlqDwWlFW6HE5BEJAo4C3geQFXLVTXP3aiMaRqVVdXMXpRKfLtW/OyswKi3U58GExRnzMgtwFI830zmq2qqiNwvIlOcZs8DMSKSBtwOHLsVeTywQUTWA+8AN6nqocY+CX/1+beHSM0q4Gdn9yQ4yHpPAtFN5yRTWFrJq1/vcTuUQNUTyAFedKYseE5EIt0Oypim8MbqDLZlF/LbAKq3U5+QhpuAqjcoWXcAACAASURBVC4BltRadm+Nx6XAjDq2mwvMbWDfnwKfehNHoHnq0510igrn0tNqjzk2gWJQfDRn9YnjhS928eMzehARGvhvKs0sBBgO3Kqqq0TkMTxfkH5fs5GI3AjcCNCtW2DekmkCW15xOX/9aDtje3Zg4qCW0eMemPcm+YH1GXl8lZ7LT8b3JDzEPrQC2U3n9OJQUTkLUjIabmxOVCaQqaqrnOcL8SQs/yXQx7KZwPfosh0UBGC9nfpYguKSf366k6iIEGaOsW9zgW5Mjw4M79aOZz5Pp6q69g1w5lSoajaQISJ9nUXnA1tcDMmYRrctu4C5X+/h2rHd6dc5sOrt1McSFBekHSxi6ZZsfnh6Em3CvbrKZvyYiHDjWT3JOFzC0tRst8MJRLcCr4nIRmAY8KDL8RjTaFSVOYu2ENUqNCDr7dTHEhQXPPtZOmHBQVx/RpLboZhmcuGAznSPac2zn9vEbY1NVdc7l2+GqOqlqnrE7ZiMaSwfbs7mq/Rc7rioL+1aB169nfpYgtLMcgrLeGf9PqaPSCC2Tbjb4ZhmEhwk3DC+B+v25rFmz+GGNzDGtHgl5VX88QNPvZ2rA7TeTn0sQWlmr369h/LKan48vofboZhmNn1EAtGtQnn2M5u4zRjTsKc/28m+vBJmTxnYIqeisASlGZVWVPHq13s4v19HesW1cTsc08xah4Vw7dhuLN2SzZ7co26HY4zxYZlHinnq051MGtKFsT1bZhkUS1Ca0bvr9pF7tJwbzrTek5Zq1ulJhAYF8cIX1otijDm+Py3Zhgj85gf93Q7FNZagNBNV5bkvdjGgSxSnt9Bs2EDHqAimDOvK/JRM8orL3Q7HGOODvtqZyweb9vOLs5OJb9fK7XBcYwlKM1mxI4e0g0X85MweLWaSHVO3n5zZg5KKKl5btdftUIwxPqayqpo5i516O2f3dDscV1mC0kye/2IXHduGM3lIV7dDMS7r1zmKM3vH8tLK3ZRVVrkdjjHGh7zxzV62ZRfyu0n9W3xpDEtQmsG27AI+//YQs8YlERZiv3IDN57Vk5zCMhatz3I7FGOMjzhytJyHP9rBuF4xTGgh9XbqY5+WzeCFL3bRKjSYa2xae+MYnxxLv85tef6LXaja9PfGGHhk2Q6Kyiq575KWU2+nPpagNLGcwjLeXZfF9BEJLW4WQHN8Ip6J27ZlF/JVeq7b4RhjXLYlq4DXVu3h2jHd6Nu5rdvh+ASvEhQRmSAi20UkTUTurmN9uIi86axfJSJJzvLRIrLe+dkgIpc5yxNF5N8islVEUkXktsY8KV8y9+s9VFRX8yOb1t7UcsnQrnSIDOOlL3e7HYoxxkWqypzFqUS3CuVXLazeTn0aTFBEJBh4ApgIDABmisiAWs1uAI6oajLwKPCQs3wzMFJVhwETgKdFJASoBO5Q1f7AWODmOvbp90orqnjNmZitp03MZmqJCA3m6tHd+HjrATIOF7sdjjHGJR9s2s+qXYdbZL2d+njTgzIaSFPVdFUtB+YBU2u1mQq87DxeCJwvIqKqxapa6SyPABRAVfer6lrncSGwFYg/tVPxPR9s3E/u0XKuH2cTs5m6XTu2OyLC3K/3uB2KMcYFJeVVPPjBVvp3iWJmC6y3Ux9vEpR4IKPG80y+n0x818ZJSPKBGAARGSMiqcAm4Oc1Ehac9UnAacCqug4uIjeKSIqIpOTk5HgRrm9QVV7+ajfJHdtwRrJNzGbq1jk6gomDOjPvm70Ul1c2vIExJqA8tWInWfmlzGmh9Xbq402CUtdvrPZtB8dto6qrVHUgMAq4R0QivttIpA3wFvA/qlpQ18FV9RmnlPrIuLg4L8L1Desy8tiYmc+s07vbaGxTrx+dkURBaSVvr93ndijGmGaUcbiYp1fs5JKhXRndo4Pb4fgcbxKUTCCxxvMEoPbkDd+1ccaYRAP/VVNeVbcCR4FBTrtQPMnJa6r69skE78teXrmbtuEhXD48we1QjI8b3q09g+OjeWnlbrvl2JgW5MElWxGBeyb2czsUn+RNgrIa6C0iPUQkDLgKWFSrzSJglvN4OrBcVdXZJgRARLoDfYHd4ulSeB7YqqqPNMaJ+JKDhaUs2bSf6SMTiAwPcTsc4+NEhOvHJZF2sIgv0+yWY2NagpU7D/Hh5mxuPieZri243k59GkxQnDEjtwBL8Qxmna+qqSJyv4hMcZo9D8SISBpwO3DsVuTxwAYRWQ+8A9ykqoeAM4DrgPNq3Ib8g0Y9Mxe9vmovFVXKD09PcjsU4ycmD+1CbJswXlppVY6NCXSVVdXMWbSFhPat+OlZLbveTn28+nqvqkuAJbWW3VvjcSkwo47t5gJz61j+BXWPW/F75ZXVvLZqL+f0jaNHbKTb4Rg/ER7iueX48X+nsSf3KN1j7G/HmEA19+s9bD9QyD+vHdHi6+3Ux2aSbWQfbt5PTmEZs8YluR2K8TPXjO1OsAivfGW3HBsTqA4VlfHIsh2c2TuWiwd2cjscn2YJSiN7eeVuesRGcnZv/7njyPiGTlER/GBwF+avzuBomd1ybEwg+su/tlFSXmX1drxgCUoj2pSZz9q9eVw3tjtBdj+7OQmzxiVRWFbJu+vtlmNjAs26vUeYn5LJDeN7kNzRZhdviCUojeillbtpHRbM9JF2a7E5OcO7tWNAlyjmfrXHbjk2JoBUVyv3LUqlY9twbj2/t9vh+AVLUBpJblEZizdmMW14AlERoW6HY/yUiHDt2O5syy5k7d4jbodjjGkk81My2JiZz29+0J82Nv2EVyxBaSTzVmdQXlnNrHHd3Q7F+Lmpw7rSNjyEuTZY1piAkF9cwV+WbmdUUnumDuvqdjh+wxKURlBVrby+ai/jesWQ3LGt2+EYPxcZHsK0EQks2ZRNblGZ2+EYY07Rox/vIK+4nNlTbGDsibAEpRGs2HGQfXklXDvWek9M47h2bDfKq6qZn5Lpdih+QUSCRWSdiLzvdizG1LR1fwGvfLWba8Z0Z2DXaLfD8SuWoDSC177eS1zbcC4cYPe0m8aR3LEtY3t24LVVe6iqtsGyXrgNz0zXxvgMVc/A2OhWodxxUR+3w/E7lqCcoswjxSzffpCrRiUSGmy/TtN4rhubROaRElbsOOh2KD5NRBKAScBzbsdiTE2LNmTxza7D3HlxP9q1DnM7HL9jn6inaN43GQhw1ehubodiAsxFAzsR1zbcBss27G/Ar4FqtwMx5pijZZU8uGQrg+KjuHJUotvh+CVLUE5BeWU181ZncF6/jsRbNUrTyEKDg5g5uhuf7sgh43Cx2+H4JBGZDBxU1TUNtLtRRFJEJCUnJ6eZojMt2ePL0zhQUMacKYMItok7T4olKKdg2ZYDHCoq45oxNjjWNI2ZoxMJEuG1VXvdDsVXnQFMEZHdwDw8FdJfrd1IVZ9R1ZGqOjIuzspQmKaVnlPE81+kM214AiO6t3c7HL/lVYIiIhNEZLuIpInI3XWsDxeRN531q0QkyVk+WkTWOz8bROQyb/fpD179eg8J7VtxVh97wzNNo0t0Ky7o35H5KRmUVVa5HY7PUdV7VDVBVZOAq4Dlqnqty2GZFkxVmbN4CxEhwdw1sa/b4fi1BhMUEQkGngAmAgOAmSIyoFazG4AjqpoMPAo85CzfDIxU1WHABOBpEQnxcp8+Le1gEV+l5zJzdDfrvjNN6rqxSRw+Ws6Hm7LdDsUY04CPtx5kxY4cbrugNx3bRrgdjl/zpgdlNJCmqumqWo6nG3VqrTZTgZedxwuB80VEVLVYVY+VZY0Ajt0v6c0+fdrrq/YSGiw2+Mk0uXG9YugRG8ncr22wbH1U9VNVnex2HKblKq2o4v73U+ndsQ2zxiW5HY7f8yZBiQcyajzPdJbV2cZJSPKBGAARGSMiqcAm4OfOem/26bNKyqtYuCaDCYO6ENsm3O1wTIALChKuGdONNXuOkJqV73Y4xpjjeOazdDIOlzBnykCbdqIRePMbrOv6Re2Zo47bRlVXqepAYBRwj4hEeLlPz459cPT9+xuzKCit5JoxdmuxaR4zRiQSHhLEG9/YYFljfFHmkWKe/DSNSYO7MC451u1wAoI3CUomUPM6RgKQdbw2IhICRAOHazZQ1a3AUWCQl/s8tp3Pjb5/ddVekju2YUyPDm6HYlqI6NahTBrShXfXZVFcXtnwBsaYZvXHDzwTGf9mUn+XIwkc3iQoq4HeItJDRMLwjJRfVKvNImCW83g6npH06mwTAiAi3YG+wG4v9+mTNu/LZ0NGHteM6WZFn0yzunp0N4rKKnl/w363QzHG1LBiRw4fbs7mlnOTbU6sRtRgguKMGbkFWIqn1sV8VU0VkftFZIrT7HkgRkTSgNuBY7cNjwc2iMh64B3gJlU9dLx9NuaJNZXXVu0hIjSIy4cnuB2KaWFGdG9P745teN0u8xjjM0orqrjvvc30jI3kp2f1dDucgBLiTSNVXQIsqbXs3hqPS4EZdWw3F5jr7T59XUFpBe+tz2LK0K5Etwp1OxzTwogIM0d34/73t7Alq4ABXaPcDsmYFu/pFenszi3m1RvGEB4S7HY4AcWGGZ+A99ZnUVxeZTPHGtdcPjyesJAg5q22XhRj3LY31xkYO6QL43vbwNjGZgnKCZj3zV4GdIliSEK026GYFqpd6zAmDe7CO2v3UVJuM8sa4xZV5b5FmwkJEn4/ya/mGfUblqB4aVNmPqlZBcwcnWiDY42rZo7uRmFZJe9vrPPGN2NMM/hoywH+vT2HX13Yh87RNmNsU7AExUvzVu8lIjSIKcP8Zj45E6BGJbWnV1ykzYlijEuKyyu5f/EW+nZqazPGNiFLULxQXF7Je+uz+MHgLjY41rju2GDZtXvz2JZd4HY4xrQ4/1iexr68Ev5w6SCbMbYJ2W/WC+9v3E9RWSUzR9vMscY3TBueQFhwEPO+yWi4sTGm0aQdLOLZz9O5fHg8o22yziZlCYoX5n2zl15xkYzs3t7tUIwBoH1kGBMHd+bttZk2WNaYZqKq3PveZlqFBnPPRJsxtqlZgtKAHQcKWbs3j6tG2cyxxrfMHN2NgtJKlmyymWWNaQ6LN+5n5c5c7ry4L3FtrVBsU7MEpQHzvskgNFi4fLgNjjW+ZUyPDvSMtcGyxjSHwtIKHnh/C4Pjo7na5sJqFpag1KO0ooq312Vy0cDOxLSxbNn4lmODZVP2HGHHgUK3wzEmoD2ybAc5RWX84dJBBAdZb3pzsASlHktTs8krrmDmKBsca3zTtBGewbLWi2JM09mYmcfLK3dzzZhuDEts53Y4LYYlKPWY900GiR1aMa5XjNuhGFOnDpFhXDyoM2+v3UdphQ2WNaaxVVZVc/dbm4htE86vJ/RzO5wWxRKU49h96Chfpedy5chEgqw7z/iwmaMSyS+pYGlqttuhGBNwXvxyN1v2FzB7ykCiImwerOZkCcpxvJmSQXCQMGNkotuhGFOvsT1jSOzQijdX25woxjSmjMPFPLJsB+f368jEQZ3dDqfF8SpBEZEJIrJdRNJE5O461oeLyJvO+lUikuQsv1BE1ojIJuff82psM9NZvlFE/iUiPlMKsqKqmgUpmZzbtyOdoqzGgvFtQUHCFSMSWbkzlz25R90Ox5iAcGzOExG4/9JBNs2ECxpMUEQkGHgCmAgMAGaKSO3SjTcAR1Q1GXgUeMhZfgi4RFUHA7OAuc4+Q4DHgHNVdQiwEbjl1E+ncXyy9SCHisq4apT1nhj/MH1kAkECC1Iy3Q7FmIDwwab9/Ht7Drdf2If4dq3cDqdF8qYHZTSQpqrpqloOzAOm1mozFXjZebwQOF9ERFXXqeqxkqupQISIhAPi/ESKJy2NAnymNOu81XvpFBXOOX3j3A7FGK90iW7FWX3iWLgmk6pqdTscY/xafkkFcxZvYVB8FNdbMUDXeJOgxAM1L25nOsvqbKOqlUA+UPvWl2nAOlUtU9UK4BfAJjyJyQDg+ROOvglk5ZWwYkcOV4xMJMSKQBk/ctWoRLILSvlsR47boRjj1x761zZyi8r48+VD7HPARd785uu68Fb7K1q9bURkIJ7LPj9znofiSVBOA7riucRzT50HF7lRRFJEJCUnp+nfeOeneHKxK2xwrPEz5/XrRExkmA2WNeYUrN59mNdX7eVHZ/RgUHy02+G0aN4kKJlAzU/rBL5/Oea7Ns74kmjgsPM8AXgH+KGq7nTaDwNQ1Z2qqsB8YFxdB1fVZ1R1pKqOjItr2ksuVdXK/NUZjE+OJbFD6yY9ljGNLSwkiMuHx/Px1gMcKipzOxxj/E5pRRV3LdxIfLtW3H5hH7fDafG8SVBWA71FpIeIhAFXAYtqtVmEZxAswHRguaqqiLQDPgDuUdUva7TfBwwQkWMZx4XA1pM9icby2bc5ZOWXcpXNHGv81JWjEqmsVt5Zu8/tUIzxO48u20H6oaM8NG0IkeEhbofT4jWYoDhjSm4BluJJIuaraqqI3C8iU5xmzwMxIpIG3A4cuxX5FiAZ+L2IrHd+OjoDZ+cAn4nIRjw9Kg826pmdhPmrM+gQGcaFAzq5HYoxJyW5Y1uGd2vHmykZeDonA5uIJIrIv0Vkq4ikishtbsdk/NO6vUd49vN0Zo5OZHxvn5n1okXzKkVU1SXAklrL7q3xuBSYUcd2DwAPHGef/wT+eSLBNqXcojI+3nqAWacnERZig6KM/7pyVCJ3vbWJtXvzGNG9vdvhNLVK4A5VXSsibYE1IrJMVbe4HZjxH2WVVfx64UY6RUVwzw/6ux2OcdgnsePd9VlUVKnNHGv83qQhXWkdFsz8FjBYVlX3q+pa53Ehnl7e2ncZGlOvxz9J49uDRTx4+WCbzt6HWIKCZ8bABSkZDE2Ipm/ntm6HY8wpaRMewuQhXVi8MYuiskq3w2k2zgzWpwGr3I3E+JPN+/J5asVOpg1P4Ny+Hd0Ox9RgCQqweV8B27ILrffEBIwrR3WjuLyKDzb6zPyHTUpE2gBvAf+jqgV1rG/W6QqMfyivrOZ/F2wgJjKMeyfXniDduM0SFDxzn4SHBHHJ0K5uh2JMoxjerR3JHdu0iDlRnHmV3gJeU9W362rTnNMVGP/xyLIdbMsu5MHLBhPd2i7t+JoWn6CUVlTx3vp9TBzUmehW9gdqAoOIcOXIRNbuzSPtYKHb4TQZp1TG88BWVX3E7XiM/1iVnsvTn+1k5uhELrA7N31Si09QlqZmU1BaaTPHmoBz2fB4QoIk0HtRzgCuA86rMZXBD9wOyvi2gtIKbp+/ge4dWvO7SXZpx1e1+JloFqRkktC+FWN71i4dZIx/i20TzgX9O/H22n3ceXG/gLx9XlW/oO5SG8Yc1+xFqWQXlLLg56fbhGw+LPDesU5A5pFivtx5iOkjEggKsvc4E3iuHJVI7tFylm874HYoxviEJZv28/bafdx8bjLDuwX8PEF+rUUnKG+t8UwHPn1EgsuRGNM0zuoTR+eoiEC/zGOMV/bnl/CbdzYxNCGaW89Ldjsc04AWm6BUVysL1mRwRq9YEtpbYUATmIKDhOkjElixI4f9+SVuh2OMayqqqrn19XVUVFbz6JXDCA1usR9/fqPF/g99nZ5L5pESZoy03hMT2K4YmUi1wsKUTLdDMcY1D3+0nZQ9R3jw8sH0jGvjdjjGCy02QZmfkkFURAgXD+zsdijGNKluMa0Z1yuG+WsyqK4O/AKCxtT2ydYDPL0inavHdGPqMKuE4C9aZIKSX1LBh5uzmTosnojQYLfDMabJXTkqkYzDJazaddjtUIxpVvvySrhjwQYGdImy2WL9TItMUN7fmEVZZbVd3jEtxsUDO9M2IoQFKTZY1rQcpRVV3PzaWiqrlCevGW5fSP2MVwmKiEwQke0ikiYid9exPlxE3nTWr3KKdiEiF4rIGhHZ5Px7Xo1twkTkGRHZISLbRGRaY51UQ+anZNKvc1sGx0c31yGNcVVEaDCXDO3Kks37KSytcDscY5qcqvLbdzazPiOPh2cMISk20u2QzAlqMEERkWDgCWAiMACYKSK1+8luAI6oajLwKPCQs/wQcImqDgZmAXNrbPNb4KCq9nH2u+JUTsRb27ML2ZCRx4yRiXhmyTamZbhiZCKlFdV8sHG/26EY0+Se/2IXb63N5H8u6M2EQV3cDsecBG96UEYDaaqarqrlwDxgaq02U4GXnccLgfNFRFR1naoeK6eaCkSISLjz/MfAnwBUtVpVD53KiXhrQUoGocHCpcOsMKBpWYYmRNO7Yxvm22UeE+BW7MjhwSVbmTioM788r7fb4ZiT5E2CEg/UfEfLdJbV2UZVK4F8oPbc8dOAdapaJiLtnGV/EJG1IrJAROqs1tSYZdIrqqp5Z90+LujfiZg24Q1vYEwAERGu+K6AYJHb4RjTJLbuL+CW19fSt3MUf71iqM0S7se8SVDq+t+tfa9ivW1EZCCeyz4/cxaFAAnAl6o6HPgKeLiugzdmmfTl2w6Se7TcCgOaFuvS0+IJDhIWrLFeFBN4Mg4XM+uFb4gMC+G5WSNpHWZ1dvyZNwlKJlDzEz0ByDpeGxEJAaKBw87zBOAd4IequtNpnwsUO8sBFgDDTyL+E7IgJYOObcM5s3dsUx/KGJ8U1zac8/p15O21+6isqnY7HGMazeGj5cx64RtKK6p45YbRxLdr5XZI5hR5k6CsBnqLSA8RCQOuAhbVarMIzyBYgOnAclVV51LOB8A9qvrlscaqqsBi4Bxn0fnAlpM+Cy8cLCjl39tzmDYigRCb4ti0YDNGJJBTWMaKHad2ydQYX1FYWsGPX1pNZl4Jz80aRZ9Obd0OyTSCBj+pnTEltwBLga3AfFVNFZH7RWSK0+x5IEZE0oDbgWO3It8CJAO/F5H1zk9HZ91dwGwR2QhcB9zRaGdVh7fX7aOqWplhhQFNC3duv47EtgljgU19bwJAYWkF17+4mk378nl85mmM7tHB7ZBMI/HqAp2qLgGW1Fp2b43HpcCMOrZ7AHjgOPvcA5x1IsGeLFVlfkoGo5LaWw0G0+KFBgdx2WnxvLRyN7lFZTZg3PitY8nJhow8/jHzNCtdEmBaxLWOtXvzSM85ygwbHGsMADNGJlJRpby7vvZwMmP8w6GiMq59bpUnObn6NCYOtrlOAk2LSFAWpGTQOiyYSfYHbAwAfTq1ZWhiOxakZOAZEmaM/9h96CjTnlrJ9gOFPH3dCJuILUAFfIJSXF7J4g1ZTBrchchwu+XMmGNmjEhgW3Yhm/cVuB2KMV5bvfsw055aSUFJBW/8dCzn969zCi0TAAI+QVmyKZuj5VVcMcou7xhT0yVDuxIeEmRzohi/oKq8+OUuZj7zNVGtQnnrF+M4rVt7t8MyTSjgE5T5KRn0iI1kZHf7QzampuhWoUwY1Jn31mdRWlHldjjGHFdecTm3vrGOOYu3cG6/jrx3yxl2w0MLENAJyu5DR/lm12FmjEywwoDG1GHGiETySypYtuWA26EYU6d/bzvIRY9+xr82Z3PnxX15+toRREWEuh2WaQYBPShj4ZpMggSmDbe5T4ypy7heMcS3a8WCNZlcMtQKaBrfsT+/hD9/uI331mfRp1MbXrh+FIPio90OyzSjgE1QqqqVhWsyObtPHJ2iItwOxxifFBQkTBuRwOPLvyUrr4SuNj24cVlRWSUvfLGLpz7dSZUqt56XzC3nJRMeEux2aKaZBewlns+/zSG7oNQKAxrTgBkjElCFt9fazLLGPfnFFTz28bec8eflPLJsB2f3ieOT28/mjov6WnLSQgVsD8qClEw6RIbZLWjGNCCxQ2tO7xnDgjWZ3Hxuso3XMs1GVVmXkcfrq/ayeEMWZZXVXNC/E7eel8zQxHZuh2dcFpAJypGj5SzbcoBrx3YnLCRgO4mMaTRXjErgV29u4JtdhxnTM8btcEwAq65WNmTm8a/N2SzZvJ+MwyVEhgUzbUQC143tTv8uUW6HaHxEQCYo763fR3lVNVeMssGxxnhjwsAu3BueyoI1mX6XoIjIBOAxIBh4TlX/7HJIpobSiiq2Zxeybu8RvkrPZdWuw+QVVxAaLJyRHMut5/bmB0O60MYm0jS1BORfxPyUTIYkRNOvs2XixnijVVgwk4d24d11WcyeMtBvPixEJBh4ArgQyARWi8giVd3ibmQti6qSU1RGxuFi9h4uZm9uCXtyj7JlfwHfHiyiqtpTTiGhfSsu7N+JcckxnNe3E9Gt7XZhc3xevQs19A1FRMKBV4ARQC5wparuFpELgT8DYUA5cKeqLq+17SKgp6oOOtWTAdi8L58t+wv4w9SBjbE7Y1qMGSMTeeObDJZs3O9PMy+PBtJUNR1AROYBUwFLUOqgqlRUKZXV1VRUKRVV1VQe+7fa86/nRykuq6SorJLi8irn30qKyqo4WlbJkeJyDhWVk1tURm5ROblHy6io+k9NJxHoHBVBv85tuaB/JwbFRzEoPpqE9q1dPHvjbxpMULz8hnIDcERVk0XkKuAh4ErgEHCJqmaJyCBgKRBfY9+XA0WNdjZ4CgOGhQQxZWh8w42NMd85LbEdT1w9nHP7xbkdyomIB2rO1Z8JjDnVnf7qzfVsyMhD4btiisc+fo/VVlRniep/lvFdm+9vU7N9XfsDrWNdHfuptYw6tqkr1soqpbL61AtDtgoNpkNkGDFtwugUFcGALlHEtAmnc1Q43WMiSezQmoT2rYgItTtvzKnxpgfFm28oU4HZzuOFwD9ERFR1XY02qUCEiISrapmItAFuB24E5p/aafxHUmwkPxqXZF2HxpwgEWHSEL+rClvXLUff+xQWkRvxvNfQrVu3BneaFBP53Ye54OkR+P/27i9ErvoM4/j3mZls0KXUGqO1TamhrhfxJuAiiF4oiFov6p82oDcNbUEvDHgbKVoxCiKIF6JCiqI3reQmuNTQai0ielNTKtbU2m412jRiAkJvREv08eKcTcZ1sju7O7vnd2afDyxzzpnfmfP+5uy8vOd//8LmrnQ6+DSe+wAABRVJREFUuXCB6rGvt50b16nh/hnnzdPffvDnneryQm1PLasa6HXEhm6HDV3R63bodcREr0Ov06HXFRu61fu9ToeJnjhzosfkRI/JjV0mN/aY3NjjjA1dup1c5RVrY5gCZZgtlJNtbJ+Q9D9gE9UelDk/Bv5q+7N6fA/wMPDJMuI+rZ9dvnWUHxcRZTsC9B+P2gIcnd/I9l5gL8D09PSiuxHuvHpqVPFFxDINcw3uMFsoC7aRdDHVYZ/b6/HtwIW29y+6cOk2SQclHTx+/PgQ4UbEOvI6MCVpq6QJ4BZgpuGYImIEhilQhtlCOdlGUg/4JvBxPb4F2A/81Pa/6/aXAZdIOgy8Clwk6eVBC7e91/a07enNm1t1bDwiVpntE8AuqvPb3gb22T7UbFQRMQrDFCjDbKHMADvr4Z8Af7JtSWcBzwN32X5trrHtJ2x/x/YFwBXAP21fubKuRMR6ZPuA7Yts/8D2A03HExGjsWiBcrotFEn3SfpR3exJYJOkWaoTX3fX03cBFwJ3S3qj/jt35L2IiIiIsTLUfVBsHwAOzJt2T9/wp8COAfPdD9y/yGcfBkZyD5SIiIgYD3lQTURERBQnBUpEREQUR55/C8SCSToOvD9E03P46j1Y2i79Kdt67c/3bbfu0rrkkbGR/pRtxXmkVQXKsCQdtD3ddByjkv6ULf0ZT+P2PaQ/ZUt/vi6HeCIiIqI4KVAiIiKiOONaoOxtOoARS3/Klv6Mp3H7HtKfsqU/84zlOSgRERHRbuO6ByUiIiJabKwKFEk7JB2S9IWk6Xnv3SVpVtI7kq5tKsblknSvpP/2PTLg+qZjWg5J19XrYFbS7sXnKJukw5L+Vq+Tg03Hs1SSnpJ0TNJbfdPOlvSipH/Vr99qMsa1ljxSvuSRsqxWHhmrAgV4C7gZeKV/oqRtVA85vBi4DnhcUnftw1uxR2xvr/8OLN68LPV3/hjwQ2AbcGu9btruqnqdtPESwaepfhP9dgMv2Z4CXuLUs7XWi+SRgiWPFOlpViGPjFWBYvtt2+8MeOsG4Fnbn9l+D5gFLl3b6ILqO5+1/a7t/wPPUq2baIjtV4CP502+AXimHn4GuHFNg2pY8kjxkkcKs1p5ZKwKlAV8F/hP3/iRelrb7JL0Zr07rY273cdlPfQz8IKkv0i6relgRuQ82x8C1K95AnllXP5/k0fKkzwywFBPMy6JpD8C3x7w1i9tP3e62QZMK+7ypYX6BjwB7KGKew/wMPDztYtuJFqxHpbocttHJZ0LvCjpH/XWRBQseSR5pDDJIwO0rkCxffUyZjsCfK9vfAtwdDQRjc6wfZP0a+B3qxzOamjFelgK20fr12OS9lPtfm57YvlI0vm2P5R0PnCs6YBGLXkkeaQkySODrZdDPDPALZI2StoKTAF/bjimJalX8JybqE7ka5vXgSlJWyVNUJ1wONNwTMsmaVLSN+aGgWto53qZbwbYWQ/vBE63R2G9SR4pQ/JIO6w4j7RuD8pCJN0EPApsBp6X9Ibta20fkrQP+DtwArjD9udNxroMD0naTrUr8zBwe7PhLJ3tE5J2AX8AusBTtg81HNZKnAfslwTVb+k3tn/fbEhLI+m3wJXAOZKOAL8CHgT2SfoF8AGwo7kI117ySNmSR8qzWnkkd5KNiIiI4qyXQzwRERHRIilQIiIiojgpUCIiIqI4KVAiIiKiOClQIiIiojgpUCIiIqI4KVAiIiKiOClQIiIiojhfAi56HuG1EsKdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 648x259.2 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_function():\n",
    "    x = np.arange(-10,10,0.05)\n",
    "    plt.figure(figsize=(9,3.6))\n",
    "\n",
    "    plt.subplot(121)\n",
    "    plt.title(\"Gaussian distribution\")\n",
    "    mu, sigma = 0, 10\n",
    "    y = lambda x : np.exp(-((x-mu)**2)/(2*sigma**2))/(sigma*np.sqrt(2*np.pi))\n",
    "    plt.plot(x, y(x))\n",
    "\n",
    "    plt.subplot(122)\n",
    "    plt.title(\"Softplus\")\n",
    "    y = np.log(1+np.exp(x))\n",
    "    plt.plot(x, y)\n",
    "    plt.show()\n",
    "\n",
    "plot_function()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "多线程又双叒叕来了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [],
   "source": [
    "def worker(worker_id, master_end, worker_end, env_name):\n",
    "    master_end.close()\n",
    "    env = gym.make(env_name)\n",
    "    env.seed(worker_id)\n",
    "    \n",
    "    while True:\n",
    "        cmd, data = worker_end.recv()\n",
    "        if cmd == 'step':\n",
    "            state, reward, done, info = env.step(data)\n",
    "            if done:\n",
    "                state = env.reset()\n",
    "            worker_end.send((state, reward, done, info))\n",
    "        elif cmd == 'reset':\n",
    "            state = env.reset()\n",
    "            worker_end.send(state)\n",
    "        elif cmd == 'reset_task':\n",
    "            state = env.reset_task()\n",
    "            worker_end.send(state)\n",
    "        elif cmd == 'close':\n",
    "            worker_end.close()\n",
    "            break\n",
    "        elif cmd == 'get_spaces':\n",
    "            worker_end.send((env.observation_space.shape[0], env.action_space.shape[0]))\n",
    "        else:\n",
    "            raise NotImplementedError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [],
   "source": [
    "class ParallelEnv:\n",
    "    def __init__(self, n_train_processes, env_name):\n",
    "        self.nenvs = n_train_processes\n",
    "        self.waiting = False\n",
    "        self.closed = False\n",
    "        self.workers = []\n",
    "        self.env_name = env_name\n",
    "        \n",
    "        self.master_ends, self.worker_ends = zip(*[mp.Pipe() for _ in range(self.nenvs)])\n",
    "        \n",
    "        for worker_id, (master_end, worker_end) in enumerate(zip(self.master_ends, self.worker_ends)):\n",
    "            p = mp.Process(target=worker, args=(worker_id, master_end, worker_end, self.env_name))\n",
    "            p.daemon = False\n",
    "            p.start()\n",
    "            self.workers.append(p)\n",
    "        for worker_end in self.worker_ends:\n",
    "            worker_end.close()\n",
    "        \n",
    "        self.master_ends[0].send(('get_spaces', None))\n",
    "        self.observation_space, self.action_space = self.master_ends[0].recv()\n",
    "        \n",
    "    def step_async(self, actions):\n",
    "        for master_end, action in zip(self.master_ends, actions):\n",
    "            master_end.send(('step', action))\n",
    "        self.waiting = True\n",
    "    \n",
    "    def step_wait(self):\n",
    "        results = [master_end.recv() for master_end in self.master_ends]\n",
    "        self.waiting = False\n",
    "        states, rewards, dones, infos = zip(*results)\n",
    "        return np.stack(states), np.stack(rewards), np.stack(dones), infos\n",
    "    \n",
    "    def reset(self):\n",
    "        for master_end in self.master_ends:\n",
    "            master_end.send(('reset', None))\n",
    "        return np.stack([master_end.recv() for master_end in self.master_ends])\n",
    "    \n",
    "    def step(self, actions):\n",
    "        self.step_async(actions)\n",
    "        return self.step_wait()\n",
    "    \n",
    "    def close(self):\n",
    "        if self.closed:\n",
    "            return\n",
    "        if self.waiting:\n",
    "            [master_end.recv() for master_end in self.master_ends]\n",
    "        for master_end in self.master_ends:\n",
    "            master_end.send(('close', None))\n",
    "        del self.workers[:]\n",
    "        self.closed = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "class Actor_critic(nn.Module):\n",
    "    def __init__(self, in_dim, out_dim, sigma):\n",
    "        super(Actor_critic, self).__init__()\n",
    "        self.actor_linear1 = nn.Linear(in_dim, 128)\n",
    "        self.critic_linear1 = nn.Linear(in_dim, 128)\n",
    "        self.linear2 = nn.Linear(128, 64)\n",
    "        self.actor_linear3 = nn.Linear(64, out_dim)\n",
    "        self.critic_linear3 = nn.Linear(64, 1)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        value_hidden = F.relu(self.linear2(F.relu(self.critic_linear1(x))))\n",
    "        value = self.critic_linear3(value_hidden)\n",
    "        actor_hidden = F.relu(self.linear2(F.relu(self.actor_linear1(x))))\n",
    "        mu = torch.tanh(self.actor_linear3(actor_hidden)) * 2\n",
    "#         sigma = F.softplus(self.sigma_linear(actor_hidden)) + 1e-6\n",
    "        dist = Normal(mu, sigma)\n",
    "        return dist, value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "画图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "code_folding": [
     0,
     30
    ]
   },
   "outputs": [],
   "source": [
    "def smooth_plot(factor, item, plot_decay):\n",
    "    item_x = np.arange(len(item))\n",
    "    item_smooth = [np.mean(item[i:i+factor]) if i > factor else np.mean(item[0:i+1])\n",
    "                  for i in range(len(item))]\n",
    "    for i in range(len(item)// plot_decay):\n",
    "        item_x = item_x[::2]\n",
    "        item_smooth = item_smooth[::2]\n",
    "    return item_x, item_smooth\n",
    "    \n",
    "def plot(episode, rewards, losses):\n",
    "    clear_output(True)\n",
    "    rewards_x, rewards_smooth = smooth_plot(10, rewards, 500)\n",
    "    losses_x, losses_smooth = smooth_plot(10, losses, 100000)\n",
    "    \n",
    "    plt.figure(figsize=(18, 10))\n",
    "    plt.subplot(211)\n",
    "    plt.title('episode %s. reward: %s'%(episode, rewards_smooth[-1]))\n",
    "    plt.plot(rewards, label=\"Rewards\", color='lightsteelblue', linewidth='1')\n",
    "    plt.plot(rewards_x, rewards_smooth, label='Smothed_Rewards', color='darkorange', linewidth='3')\n",
    "    plt.legend(loc='best')\n",
    "    \n",
    "    plt.subplot(212)\n",
    "    plt.title('Losses')\n",
    "    plt.plot(losses,label=\"Losses\",color='lightsteelblue',linewidth='1')\n",
    "    plt.plot(losses_x, losses_smooth, \n",
    "             label=\"Smoothed_Losses\",color='darkorange',linewidth='3')\n",
    "    plt.legend(loc='best')\n",
    "    \n",
    "    plt.show()\n",
    "    \n",
    "def test_env():\n",
    "    state = env.reset()\n",
    "    done = False\n",
    "    total_reward = 0\n",
    "    while not done:\n",
    "        state = torch.FloatTensor(state).reshape(-1, 3).to(device)\n",
    "        log_prob, _ = model(state)\n",
    "        next_state, reward, done, _ = env.step(log_prob.sample().cpu().numpy())\n",
    "        state = next_state\n",
    "        total_reward += reward\n",
    "    return total_reward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "def gae_compute(next_value, rewards, masks, values, gamma=0.99, tau=0.95):\n",
    "    td_target = next_value\n",
    "    delta_list = []\n",
    "    for idx in reversed(range(len(rewards))):\n",
    "        td_target = td_target * gamma * masks[idx] + rewards[idx]\n",
    "        delta = td_target - values[idx]\n",
    "        delta_list.insert(0, delta)\n",
    "    \n",
    "    advantage = 0\n",
    "    advantage_list = []\n",
    "    for idx in reversed(range(len(delta_list))):\n",
    "        advantage = delta_list[idx] + advantage * gamma * tau\n",
    "        advantage_list.insert(0, advantage)\n",
    "    return advantage_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PPO训练更新"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "def ppo_iter(states, actions, log_probs, advantages):\n",
    "    batch_size = actions.size(0)\n",
    "    for _ in range(batch_size // mini_batch_size):\n",
    "        ids = np.random.choice(batch_size, mini_batch_size, replace=False)\n",
    "        yield states[ids, :], actions[ids, :], log_probs[ids, :], advantages[ids, :]\n",
    "\n",
    "def ppo_train(states, actions, log_probs, advantages, clip_param=0.2):\n",
    "    losses = []\n",
    "    for _ in range(ppo_epochs):\n",
    "        for state, action, old_log_probs, advantage in ppo_iter(states, actions, log_probs, advantages):\n",
    "            dist, _ = model(state)\n",
    "            entropy = dist.entropy().mean()\n",
    "            new_log_probs = dist.log_prob(action)\n",
    "            \n",
    "            ratio = (new_log_probs - old_log_probs).exp() \n",
    "            sub1 = ratio * advantage\n",
    "            sub2 = torch.clamp(ratio, 1.0-clip_param, 1.0+clip_param) * advantage\n",
    "            actor_loss = - torch.min(sub1, sub2).mean()\n",
    "            critic_loss = advantage.pow(2).mean()\n",
    "            \n",
    "            loss = 0.5 * critic_loss + actor_loss - 0.001 * entropy\n",
    "            losses.append(loss.item())\n",
    "            \n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "        \n",
    "        old_model.load_state_dict(model.state_dict())\n",
    "            \n",
    "    return round(mean(losses),2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "code_folding": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Process Process-11:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "Process Process-16:\n",
      "Process Process-1:\n",
      "Process Process-3:\n",
      "Process Process-10:\n",
      "Process Process-7:\n",
      "Process Process-8:\n",
      "Process Process-15:\n",
      "Process Process-4:\n",
      "Process Process-6:\n",
      "Process Process-12:\n",
      "Process Process-9:\n",
      "Process Process-5:\n",
      "Process Process-13:\n",
      "Process Process-14:\n",
      "Process Process-2:\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "KeyboardInterrupt\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 258, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "KeyboardInterrupt\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"<ipython-input-3-0fcf21e5fd84>\", line 7, in worker\n",
      "    cmd, data = worker_end.recv()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "KeyboardInterrupt\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "KeyboardInterrupt\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 250, in recv\n",
      "    buf = self._recv_bytes()\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "KeyboardInterrupt\n",
      "  File \"/opt/conda/lib/python3.6/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n"
     ]
    }
   ],
   "source": [
    "## hyperparameters ##\n",
    "\n",
    "num_envs = 16\n",
    "env_name = \"Pendulum-v0\"\n",
    "ppo_epochs = 16\n",
    "mini_batch_size = 5\n",
    "max_epoch = 10000\n",
    "max_timesteps = 5\n",
    "sigma = torch.tensor(0.5, dtype=torch.float32).to(device)\n",
    "\n",
    "## hyperparameters ##\n",
    "\n",
    "envs = ParallelEnv(num_envs, env_name)\n",
    "state_space = envs.observation_space\n",
    "action_space = envs.action_space\n",
    "\n",
    "env = gym.make(env_name)\n",
    "\n",
    "model = Actor_critic(state_space, action_space, sigma).to(device)\n",
    "optimizer = optim.Adam(model.parameters(), lr=1e-3)\n",
    "old_model = Actor_critic(state_space, action_space, sigma).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "code_folding": []
   },
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-aaa4dc840f37>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     30\u001b[0m         \u001b[0mtest_reward\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtest_env\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m         \u001b[0mtest_rewards\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtest_reward\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 32\u001b[0;31m         \u001b[0mplot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepoch\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest_rewards\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     33\u001b[0m \u001b[0;31m#         soft = lambda loss : np.mean(loss[-100:]) if len(loss)>=100 else np.mean(loss)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[0;31m#         writer.add_scalar(\"Test_Rewards\", np.array(soft(test_rewards)), epoch)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-6-18a61c9c24c7>\u001b[0m in \u001b[0;36mplot\u001b[0;34m(episode, rewards, losses)\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mplot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepisode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrewards\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlosses\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0mclear_output\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m     \u001b[0mrewards_x\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrewards_smooth\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msmooth_plot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrewards\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m200\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     13\u001b[0m     \u001b[0mlosses_x\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlosses_smooth\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msmooth_plot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlosses\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m10000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-6-18a61c9c24c7>\u001b[0m in \u001b[0;36msmooth_plot\u001b[0;34m(factor, item, plot_decay)\u001b[0m\n\u001b[1;32m      2\u001b[0m     \u001b[0mitem_x\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m     item_smooth = [np.mean(item[i:i+factor]) if i > factor else np.mean(item[0:i+1])\n\u001b[0;32m----> 4\u001b[0;31m                   for i in range(len(item))]\n\u001b[0m\u001b[1;32m      5\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m//\u001b[0m \u001b[0mplot_decay\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m         \u001b[0mitem_x\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mitem_x\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-6-18a61c9c24c7>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m      2\u001b[0m     \u001b[0mitem_x\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m     item_smooth = [np.mean(item[i:i+factor]) if i > factor else np.mean(item[0:i+1])\n\u001b[0;32m----> 4\u001b[0;31m                   for i in range(len(item))]\n\u001b[0m\u001b[1;32m      5\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m//\u001b[0m \u001b[0mplot_decay\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m         \u001b[0mitem_x\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mitem_x\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<__array_function__ internals>\u001b[0m in \u001b[0;36mmean\u001b[0;34m(*args, **kwargs)\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/numpy/core/fromnumeric.py\u001b[0m in \u001b[0;36mmean\u001b[0;34m(a, axis, dtype, out, keepdims)\u001b[0m\n\u001b[1;32m   3255\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3256\u001b[0m     return _methods._mean(a, axis=axis, dtype=dtype,\n\u001b[0;32m-> 3257\u001b[0;31m                           out=out, **kwargs)\n\u001b[0m\u001b[1;32m   3258\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3259\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/numpy/core/_methods.py\u001b[0m in \u001b[0;36m_mean\u001b[0;34m(a, axis, dtype, out, keepdims)\u001b[0m\n\u001b[1;32m    133\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    134\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_mean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeepdims\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 135\u001b[0;31m     \u001b[0marr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0masanyarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    136\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    137\u001b[0m     \u001b[0mis_float16_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/numpy/core/_asarray.py\u001b[0m in \u001b[0;36masanyarray\u001b[0;34m(a, dtype, order)\u001b[0m\n\u001b[1;32m    136\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    137\u001b[0m     \"\"\"\n\u001b[0;32m--> 138\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0morder\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0morder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msubok\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    139\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    140\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "test_rewards = []\n",
    "loss_list = []\n",
    "state = envs.reset()\n",
    "\n",
    "for epoch in range(max_epoch):\n",
    "    states, actions, rewards, masks, log_probs, values = [], [], [], [], [], []\n",
    "    \n",
    "    for _ in range(max_timesteps):\n",
    "        dist, value = old_model(torch.FloatTensor(state).to(device))\n",
    "        action = dist.sample()\n",
    "        \n",
    "        next_state, reward, done, _ = envs.step(action.cpu().numpy())\n",
    "        \n",
    "        states.append(torch.FloatTensor(state).to(device))\n",
    "        actions.append(action)\n",
    "        rewards.append(torch.FloatTensor(reward).unsqueeze(1).to(device))\n",
    "        masks.append(torch.FloatTensor(1 - done).unsqueeze(1).to(device))\n",
    "        log_probs.append(dist.log_prob(action))\n",
    "        values.append(value)\n",
    "        \n",
    "        state = next_state\n",
    "        \n",
    "    _, next_value = model(torch.FloatTensor(next_state).to(device))\n",
    "    advantages = gae_compute(next_value, rewards, masks, values)\n",
    "    loss = ppo_train(torch.cat(states).detach(),torch.cat(actions).detach(), \n",
    "                     torch.cat(log_probs).detach(), torch.cat(advantages).detach())\n",
    "    loss_list.append(loss)\n",
    "    \n",
    "    if epoch % 1 == 0:\n",
    "        test_reward = np.mean([test_env() for _ in range(10)])\n",
    "        test_rewards.append(test_reward)\n",
    "        plot(epoch + 1, test_rewards, loss_list)\n",
    "#         soft = lambda loss : np.mean(loss[-100:]) if len(loss)>=100 else np.mean(loss)\n",
    "#         writer.add_scalar(\"Test_Rewards\", np.array(soft(test_rewards)), epoch)\n",
    "#         writer.add_scalar(\"Value_Losses\", np.array(soft(loss_list)), epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 1)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state_1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-24-f8031d63431e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m     \u001b[0mimg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'rgb_array'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# just update the data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m     \u001b[0mdisplay\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisplay\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgcf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m     \u001b[0mdisplay\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclear_output\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/IPython/core/display.py\u001b[0m in \u001b[0;36mdisplay\u001b[0;34m(include, exclude, metadata, transient, display_id, *objs, **kwargs)\u001b[0m\n\u001b[1;32m    302\u001b[0m             \u001b[0mpublish_display_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmetadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmetadata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    303\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 304\u001b[0;31m             \u001b[0mformat_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmd_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minclude\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minclude\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexclude\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mexclude\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    305\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mformat_dict\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    306\u001b[0m                 \u001b[0;31m# nothing to display (e.g. _ipython_display_ took over)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/IPython/core/formatters.py\u001b[0m in \u001b[0;36mformat\u001b[0;34m(self, obj, include, exclude)\u001b[0m\n\u001b[1;32m    178\u001b[0m             \u001b[0mmd\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    179\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 180\u001b[0;31m                 \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mformatter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    181\u001b[0m             \u001b[0;32mexcept\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    182\u001b[0m                 \u001b[0;31m# FIXME: log the exception\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m</opt/conda/lib/python3.6/site-packages/decorator.py:decorator-gen-9>\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, obj)\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/IPython/core/formatters.py\u001b[0m in \u001b[0;36mcatch_format_error\u001b[0;34m(method, self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    222\u001b[0m     \u001b[0;34m\"\"\"show traceback on failed format call\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    223\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 224\u001b[0;31m         \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    225\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    226\u001b[0m         \u001b[0;31m# don't warn on NotImplementedErrors\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/IPython/core/formatters.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, obj)\u001b[0m\n\u001b[1;32m    339\u001b[0m                 \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    340\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 341\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mprinter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    342\u001b[0m             \u001b[0;31m# Finally look for special method names\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    343\u001b[0m             \u001b[0mmethod\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_real_method\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprint_method\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/IPython/core/pylabtools.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(fig)\u001b[0m\n\u001b[1;32m    242\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    243\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;34m'png'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mformats\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 244\u001b[0;31m         \u001b[0mpng_formatter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfor_type\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mFigure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mfig\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mprint_figure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'png'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    245\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;34m'retina'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mformats\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m'png2x'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mformats\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    246\u001b[0m         \u001b[0mpng_formatter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfor_type\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mFigure\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mfig\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mretina_figure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/IPython/core/pylabtools.py\u001b[0m in \u001b[0;36mprint_figure\u001b[0;34m(fig, fmt, bbox_inches, **kwargs)\u001b[0m\n\u001b[1;32m    126\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    127\u001b[0m     \u001b[0mbytes_io\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mBytesIO\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 128\u001b[0;31m     \u001b[0mfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcanvas\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprint_figure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbytes_io\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    129\u001b[0m     \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbytes_io\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    130\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfmt\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'svg'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/backend_bases.py\u001b[0m in \u001b[0;36mprint_figure\u001b[0;34m(self, filename, dpi, facecolor, edgecolor, orientation, format, bbox_inches, **kwargs)\u001b[0m\n\u001b[1;32m   2080\u001b[0m                     \u001b[0morientation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0morientation\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2081\u001b[0m                     \u001b[0mbbox_inches_restore\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0m_bbox_inches_restore\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2082\u001b[0;31m                     **kwargs)\n\u001b[0m\u001b[1;32m   2083\u001b[0m             \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2084\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mbbox_inches\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mrestore_bbox\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py\u001b[0m in \u001b[0;36mprint_png\u001b[0;34m(self, filename_or_obj, metadata, pil_kwargs, *args, **kwargs)\u001b[0m\n\u001b[1;32m    525\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    526\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 527\u001b[0;31m             \u001b[0mFigureCanvasAgg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    528\u001b[0m             \u001b[0mrenderer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_renderer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    529\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mcbook\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_setattr_cm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdpi\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdpi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py\u001b[0m in \u001b[0;36mdraw\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    386\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrenderer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_renderer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcleared\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    387\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mRendererAgg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 388\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    389\u001b[0m             \u001b[0;31m# A GUI class may be need to update a window using this draw, so\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    390\u001b[0m             \u001b[0;31m# don't forget to call the superclass.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/artist.py\u001b[0m in \u001b[0;36mdraw_wrapper\u001b[0;34m(artist, renderer, *args, **kwargs)\u001b[0m\n\u001b[1;32m     36\u001b[0m                 \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 38\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0martist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrenderer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     39\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     40\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0martist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_agg_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/figure.py\u001b[0m in \u001b[0;36mdraw\u001b[0;34m(self, renderer)\u001b[0m\n\u001b[1;32m   1707\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpatch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1708\u001b[0m             mimage._draw_list_compositing_images(\n\u001b[0;32m-> 1709\u001b[0;31m                 renderer, self, artists, self.suppressComposite)\n\u001b[0m\u001b[1;32m   1710\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1711\u001b[0m             \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose_group\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'figure'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/image.py\u001b[0m in \u001b[0;36m_draw_list_compositing_images\u001b[0;34m(renderer, parent, artists, suppress_composite)\u001b[0m\n\u001b[1;32m    133\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mnot_composite\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhas_images\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    134\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0ma\u001b[0m \u001b[0;32min\u001b[0m \u001b[0martists\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 135\u001b[0;31m             \u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    136\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    137\u001b[0m         \u001b[0;31m# Composite any adjacent images together\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/artist.py\u001b[0m in \u001b[0;36mdraw_wrapper\u001b[0;34m(artist, renderer, *args, **kwargs)\u001b[0m\n\u001b[1;32m     36\u001b[0m                 \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 38\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0martist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrenderer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     39\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     40\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0martist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_agg_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/axes/_base.py\u001b[0m in \u001b[0;36mdraw\u001b[0;34m(self, renderer, inframe)\u001b[0m\n\u001b[1;32m   2643\u001b[0m             \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstop_rasterizing\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2644\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2645\u001b[0;31m         \u001b[0mmimage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_draw_list_compositing_images\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0martists\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2646\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2647\u001b[0m         \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose_group\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'axes'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/image.py\u001b[0m in \u001b[0;36m_draw_list_compositing_images\u001b[0;34m(renderer, parent, artists, suppress_composite)\u001b[0m\n\u001b[1;32m    133\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mnot_composite\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhas_images\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    134\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0ma\u001b[0m \u001b[0;32min\u001b[0m \u001b[0martists\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 135\u001b[0;31m             \u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    136\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    137\u001b[0m         \u001b[0;31m# Composite any adjacent images together\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/artist.py\u001b[0m in \u001b[0;36mdraw_wrapper\u001b[0;34m(artist, renderer, *args, **kwargs)\u001b[0m\n\u001b[1;32m     36\u001b[0m                 \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 38\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0martist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrenderer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     39\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     40\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0martist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_agg_filter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/image.py\u001b[0m in \u001b[0;36mdraw\u001b[0;34m(self, renderer, *args, **kwargs)\u001b[0m\n\u001b[1;32m    617\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    618\u001b[0m             im, l, b, trans = self.make_image(\n\u001b[0;32m--> 619\u001b[0;31m                 renderer, renderer.get_image_magnification())\n\u001b[0m\u001b[1;32m    620\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mim\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    621\u001b[0m                 \u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw_image\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mim\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/image.py\u001b[0m in \u001b[0;36mmake_image\u001b[0;34m(self, renderer, magnification, unsampled)\u001b[0m\n\u001b[1;32m    872\u001b[0m         return self._make_image(\n\u001b[1;32m    873\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_A\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbbox\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtransformed_bbox\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maxes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbbox\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmagnification\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 874\u001b[0;31m             unsampled=unsampled)\n\u001b[0m\u001b[1;32m    875\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    876\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_check_unsampled_image\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrenderer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/image.py\u001b[0m in \u001b[0;36m_make_image\u001b[0;34m(self, A, in_bbox, out_bbox, clip_bbox, magnification, unsampled, round_to_pixel_border)\u001b[0m\n\u001b[1;32m    505\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    506\u001b[0m                 \u001b[0;31m#resample rgb channels\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 507\u001b[0;31m                 \u001b[0mA\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_rgb_to_rgba\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mA\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m...\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    508\u001b[0m                 _image.resample(\n\u001b[1;32m    509\u001b[0m                     \u001b[0mA\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/matplotlib/image.py\u001b[0m in \u001b[0;36m_rgb_to_rgba\u001b[0;34m(A)\u001b[0m\n\u001b[1;32m    167\u001b[0m     \"\"\"\n\u001b[1;32m    168\u001b[0m     \u001b[0mrgba\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mA\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mA\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mA\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 169\u001b[0;31m     \u001b[0mrgba\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mA\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    170\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mrgba\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muint8\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    171\u001b[0m         \u001b[0mrgba\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m255\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAD8CAYAAAB3lxGOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAARF0lEQVR4nO3df4xV5Z3H8feHnzrFVBkHQ/nh0EiMjVpqp5bWjVZAY5UqacXqNko2ZPlj3cRGkyrZZDdN9g+MSTEmbbPsaoqbpv6qFpiaGOVHm01X6lDURYkypWydQAQiaFcQRb77x32GXobLzJ2Zc+fcmefzSk7ueZ7zzLnfgbmfec6Pe0cRgZnla1zZBZhZuRwCZplzCJhlziFgljmHgFnmHAJmmWtICEi6QdJbkrolPdCI5zCzYqjo+wQkjQfeBq4DeoBXgDsi4s1Cn8jMCtGImcCVQHdE7I6Ij4EngFsa8DxmVoAJDdjnDOCdqnYP8NX+vuD888+P9vb2BpRiZr22bdt2MCLa+vY3IgRUo++0Yw5JK4AVALNnz6arq6sBpZhZL0n/W6u/EYcDPcCsqvZMYG/fQRGxJiI6IqKjre20cDKzEdKIEHgFmCtpjqRJwO3A+gY8j5kVoPDDgYg4LukfgReA8cBjEfFG0c9jZsVoxDkBIuJ54PlG7NvMiuU7Bs0y5xAwy5xDwCxzDgGzzDkEzDLnEDDLnEPALHMOAbPMOQTMMucQMMucQ8Ascw4Bs8w5BMwy5xAwy5xDwCxzDgGzzDkEzDLnEDDLnEPALHMOAbPMOQTMMucQMMucQ8Ascw4Bs8w5BMwy5xAwy5xDwCxzDgGzzDkEzDLnEDDLnEPALHMOAbPMOQTMMucQMMucQ8AscwOGgKTHJO2XtKOqb6qkFyXtSo/npX5JekRSt6TXJV3RyOLNbPjqmQn8DLihT98DwMaImAtsTG2AbwJz07IC+GkxZZpZowwYAhHxW+C9Pt23AGvT+lpgSVX/41HxMnCupOlFFWtmxRvqOYELImIfQHqclvpnAO9UjetJfaeRtEJSl6SuAwcODLEMMxuuok8MqkZf1BoYEWsioiMiOtra2gouw8zqNdQQeLd3mp8e96f+HmBW1biZwN6hl2dmjTbUEFgPLEvry4B1Vf13pasE84H3ew8bzKw5TRhogKRfAN8AzpfUA/wLsAp4StJy4M/A0jT8eeBGoBs4AvxdA2o2swINGAIRcccZNi2sMTaAu4dblJmNHN8xaJY5h4BZ5hwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWeYGDAFJsyRtlrRT0huS7kn9UyW9KGlXejwv9UvSI5K6Jb0u6YpGfxNWjnHjxjFnzhxWrVrFjh07yi7HhqiemcBx4L6IuASYD9wt6QvAA8DGiJgLbExtgG8Cc9OyAvhp4VVbU4gI9uzZw8qVK7nsssuQdMqycePGsku0OgwYAhGxLyL+kNb/AuwEZgC3AGvTsLXAkrR+C/B4VLwMnCtpeuGVW+kiouYiCYBFixYhiaNHj5ZcqfVnUOcEJLUDXwK2AhdExD6oBAUwLQ2bAbxT9WU9qc8yceLECSKCVatWAdDS0sK4cT791Kzq/p+RNAX4JfD9iPigv6E1+qLG/lZI6pLUdeDAgXrLsFHk/vvvJyKYNGnSKTMEay51hYCkiVQC4OcR8Wzqfrd3mp8e96f+HmBW1ZfPBPb23WdErImIjojoaGtrG2r9NgocO3aMiODxxx8/eb7Amkc9VwcEPArsjIgfVW1aDyxL68uAdVX9d6WrBPOB93sPGyxvd955J0uWVE4dOQiaRz0zgauAO4EFkl5Ny43AKuA6SbuA61Ib4HlgN9AN/DvwD8WXbaPVc889xyuvvAJAe3t7ucUYABMGGhAR/0Xt43yAhTXGB3D3MOuyMayjo+PkOQJJVH5krCw+ZWul+c1vfgP40KBsDgErzdVXX82MGb56XDaHgJWqp6eH66+/3rOBEjkErHQvvPACAFdc4beZlMEhYE1j+/btZZeQJYeANYUPP/wQgM997nMlV5Ifh4A1hZaWFlpbW9m3b58vGY4wh4A1jYMHDwIwZcqUkivJi0PAms6RI0fKLiErDoExbrS9hffNN98su4TsjK6fEBu0iODEiRNll1G3Sy65pOwSsjPgewds9Lr77spbOC644AKK+syGbd/61intL2/YUMh+rTyeCYxhP/nJT4C/nnAbrr4BcKa+IvzqV79qyH7tdA6BMerw4cOF7q+/F3vRQTBt2jTuu+++QvdpZ+YQGKOmTp16Svvee+8d8r6qX+QdnZ2nLLXGDNc111zD7t27C9uf9c8hMEb1veFm9erVJVUyeAsWLCi7hKw4BMaw3iB48sknAU5+os9gNOqYvz+9txDbyPDVgTHopZdeOmUm8Kc//YmIYPz48Xz66aclVlafLVu2lF1CVhwCY9CiRYtOab/88ssAhQRA1+LFNfuLvFS4detWLrzwwsL2Z/1zCGRgqJfbyjgUADhw4ADPPPNMKc+dI58TsGFrxA1D3/nOdwrfp9XmEBjj/LHeNhCHwBi3YYi/pcs6FHj77bdLed6cOQTGuEsvvbSh+y/6UODiiy8udH82MIdABlpbWznnnHPqHl/WLKBXa2trqc+fG18dyMDBgwdHxUd6z5pV+Tu2Rb3hyerjmUBG6gmCwcwCijwUOHr0KD09PUyfPr2wfVp9HAKZmDx5MlB5sTWjlpYWAPbuPe2v2FuDOQQy8dFHHwF/fbENVyPuDbj88ssL36cNzCGQkXnz5pVdQk0333wzAK+99lrJleTJJwYzsn37djo6Os7458DLuCowZ84c9uzZ4781UCLPBDLT1dUFDO9TiIs6FNi0aRN79uwpZF82dA6BDC1ZsuTkW4t7lTELWLhwIXD6B6DYyHIIZOi5557j448/5sSJE0ycOHFQX1vELKCrqwtJTJs2zQHQBBwCmZo4cSIRwfHjx0f0RqKlS5fyla98BYB33313xJ7XzmzAEJB0lqTfS3pN0huSfpj650jaKmmXpCclTUr9k1O7O21vb+y3YMOxdOlSoPIBotvfe6/fscOdBTz11FMnPyfAM4DmUc9M4BiwICK+CMwDbpA0H3gQWB0Rc4FDwPI0fjlwKCIuAlancdak7j96lK7Fi7li6lT+/ne/o6Ozk7c/+KDQ5zjrrLOQxHe/+10iwgHQZAYMgaj4v9ScmJYAFgC9H/+yFliS1m9JbdL2hRoNN65nbs3Xv37yo8P+9re/paOzkxPDfLE+9NBDSOLYsWNnvCxp5avrPgFJ44FtwEXAj4E/Aocj4nga0gPMSOszgHcAIuK4pPeBVuBgn32uAFYAzJ49e3jfhRWma/Fivr15M3/+8EOu/PWvAThv0iRevP76uvcxbty4U17wR44c4eyzzy68VitGXScGI+LTiJgHzASuBGr91cje//Vav/VP+xUQEWsioiMiOtra2uqt1wp0psuCz157LV2LFzP7M58B4NDHH9PR2Ymkk8v48eOZMmXKKX29S28ArFu3johwADS5Qd0xGBGHJW0B5gPnSpqQZgMzgd53fvQAs4AeSROAzwL9n3GypvTstdeeXP/yhg0cOXKEzs5Onn76aV566SU++eQTLr74YpYtW8ZNN93ke/9HqQFDQFIb8EkKgLOBRVRO9m0GbgWeAJYB69KXrE/t/07bN4UPBpvOUN4y3NLSwm233cZtt93WqLKsBPXMBKYDa9N5gXHAUxHRKelN4AlJ/wpsBx5N4x8F/lNSN5UZwO0NqNvMCjJgCETE68CXavTvpnJ+oG//R8DSQqqzhijrg0OsOfmOQbPMOQTsjDwLyINDwCxzDoHMlP1x4tZ8HAJmmXMIZMRXBawWh4BZ5hwCmfAswM7EIWCWOYeAncKzgPw4BDLgy4LWH4eAWeYcAnaSDwXy5BAY43woYANxCBjgWUDOHAJjmGcBVg+HgHkWkDmHgFnmHAJmmXMIZM6HAuYQMMucQ2AMG+i3vGcBBg6BMe/LGzbUfLE7AKzXoP4MmY1eftHbmXgmYJY5h4BZ5hwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWebqDgFJ4yVtl9SZ2nMkbZW0S9KTkial/smp3Z22tzemdDMrwmBmAvcAO6vaDwKrI2IucAhYnvqXA4ci4iJgdRpnZk2qrhCQNBO4CfiP1BawAHgmDVkLLEnrt6Q2afvCNN7MmlC9M4GHgR8AJ1K7FTgcEcdTuweYkdZnAO8ApO3vp/Fm1oQGDAFJi4H9EbGturvG0KhjW/V+V0jqktR14MCBuoo1s+LVMxO4CrhZ0h7gCSqHAQ8D50rq/TyCmcDetN4DzAJI2z8LvNd3pxGxJiI6IqKjra1tWN+EmQ3dgCEQESsjYmZEtAO3A5si4nvAZuDWNGwZsC6tr09t0vZNEXHaTMDMmsNw7hO4H7hXUjeVY/5HU/+jQGvqvxd4YHglmlkjDerjxSJiC7Alre8Grqwx5iNgaQG1mdkI8B2DZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlThFRdg1I+gvwVtl1DML5wMGyi6jTaKoVRle9o6lWgAsjoq1v54QyKqnhrYjoKLuIeknqGi31jqZaYXTVO5pq7Y8PB8wy5xAwy1yzhMCasgsYpNFU72iqFUZXvaOp1jNqihODZlaeZpkJmFlJSg8BSTdIektSt6QHmqCexyTtl7Sjqm+qpBcl7UqP56V+SXok1f66pCtKqHeWpM2Sdkp6Q9I9zVqzpLMk/V7Sa6nWH6b+OZK2plqflDQp9U9O7e60vX2kaq2qebyk7ZI6m73WoSo1BCSNB34MfBP4AnCHpC+UWRPwM+CGPn0PABsjYi6wMbWhUvfctKwAfjpCNVY7DtwXEZcA84G7079hM9Z8DFgQEV8E5gE3SJoPPAisTrUeApan8cuBQxFxEbA6jRtp9wA7q9rNXOvQRERpC/A14IWq9kpgZZk1pTragR1V7beA6Wl9OpX7GgD+Dbij1rgSa18HXNfsNQMtwB+Ar1K54WZC358J4AXga2l9QhqnEaxxJpUAXQB0AmrWWoezlH04MAN4p6rdk/qazQURsQ8gPU5L/U1Vf5qCfgnYSpPWnKbXrwL7gReBPwKHI+J4jXpO1pq2vw+0jlStwMPAD4ATqd1K89Y6ZGWHgGr0jabLFU1Tv6QpwC+B70fEB/0NrdE3YjVHxKcRMY/Kb9krgUv6qae0WiUtBvZHxLbq7n7qaZqfhcEqOwR6gFlV7ZnA3pJq6c+7kqYDpMf9qb8p6pc0kUoA/Dwink3dTV1zRBwGtlA5j3GupN5b2KvrOVlr2v5Z4L0RKvEq4GZJe4AnqBwSPNyktQ5L2SHwCjA3nXGdBNwOrC+5plrWA8vS+jIqx929/XelM+7zgfd7p+AjRZKAR4GdEfGjqk1NV7OkNknnpvWzgUVUTrptBm49Q62938OtwKZIB92NFhErI2JmRLRT+bncFBHfa8Zah63skxLAjcDbVI4N/6kJ6vkFsA/4hEq6L6dybLcR2JUep6axonJ144/A/wAdJdT7N1Smna8Dr6blxmasGbgc2J5q3QH8c+r/PPB7oBt4Gpic+s9K7e60/fMl/Ux8A+gcDbUOZfEdg2aZK/twwMxK5hAwy5xDwCxzDgGzzDkEzDLnEDDLnEPALHMOAbPM/T81T7Se7BYlBwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython import display\n",
    "\n",
    "env = gym.make(env_name)\n",
    "state_1 = env.reset()\n",
    "img = plt.imshow(env.render(mode='rgb_array')) # only call this once\n",
    "for _ in range(1000):\n",
    "    img.set_data(env.render(mode='rgb_array')) # just update the data\n",
    "    display.display(plt.gcf())\n",
    "    display.clear_output(wait=True)\n",
    "    \n",
    "    prob, value = old_model(torch.FloatTensor(state_1).reshape(1,-1).to(device))\n",
    "    action = prob.sample().cpu().numpy()\n",
    "    next_state, _, done, _ = env.step(action)\n",
    "    if done: \n",
    "        state_1 = env.reset()\n",
    "    state_1 = next_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PPO Baselines:\n",
    "<img src=\"../assets/PPO_baseline.png\"></img>\n",
    "### Test_Rewards:\n",
    "<img src=\"../assets/PPO_Test_Rewards.png\" width=100%></img>\n",
    "### Value_Losses:\n",
    "<img src=\"../assets/PPO_Value_Losses.png\"></img>"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
